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Preface 


The  purpose  of  this  research  was  to  generate  cost  and 
schedule  benchmarks  in  percent  for  Department  of  Defense 
contracts  for  48  collectively  exhaustive  and  mutually 
exclusive  categories  of  contracts.  Cost  and  schedule 
veirieuice  benchmarks  that  are  tailored  to  specific  categories 
of  contracts  are  needed  to  assist  program  meuiagers  in 
determining  the  cost  euid  schedule  performance  of  their 
current  programs. 

Hypothesis  testing  was  accon^lished  across  categories  of 
contracts  to  determine  if  various  benchmarks  were 
statistically  significantly  different.  If  a  given  cost  or 
schedule  benchmark  was  found  to  be  statistically 
representative  of  a  specific  category  of  contract  then  that 
benchmark  would  provide  program  managers  with  a  sound 
historical  standard  against  which  they  could  evaluate  their 
current  programs. 

Special  thanks  to  our  primary  thesis  advisor.  Dr.  Dave 
Christensen,  for  his  advice  and  guidance  throughout  the 
entire  process,  in  addition,  we  greatly  appreciated  the  many 
hours  that  Professor  Dan  Reynolds  unselfishly  spent 
counseling  us  on  the  finer  points  of  ANOVA  nested  emalyses. 


Glenn  Buchfeller  and  Donald  Kehl 
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Abstract 

Managers  oj:  Department  of  Defense  programs  would  benefit 
frcmi  having  cost  and  schedule  benchmarks  that  are  based  on 
tho  historical  performance  of  similar  programs.  This 
research  generated  cost  and  schedule  variance  in  percent 
benchmarks  for  48  collectively  exhaustive  and  mutually 
exclusive  categories  of  DoD  contracts. 

The  analysis  of  variance  (ANOVA)  nested  design 
methodology  was  used  to  compare  these  cost  and  schedule 
benchmarks  across  related  categories  of  contracts  to 
determine  if  the  benchmarks  were  statistically  significantly 
different.  Such  statistical  difference  would  ensure  program 
managers  had  a  very  specific  tool  tailored  to  their  unique 
needs. 

Due  to  some  relatively  small  sao^le  sizes  in  the  study, 
along  with  the  rather  large  standard  deviations  associated 
with  those  samples,  the  majority  of  benchmarks  did  not  prove 
to  be  statistically  significantly  different.  As  a  result, 
most  benchmarks  do  not  uncategorically  describe  one,  and  only 
one,  category  of  contract.  Thus,  progreun  managers  must 
exercise  caution  when  drawing  conclusions  about  how  the  cost 
and  schedule  performance  of  their  current  programs  con^ares 
to  the  historical  average. 

A  few  years  from  now,  as  the  number  of  contracts 
included  in  the  DAES  database  grows  larger,  a  greater  number 
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of  cost  and  schedule  benchmarks  that  test  statistically 
significantly  different  should  be  able  to  be  calculated. 


COST  AND  SCHEDULE  BENCHMARKS 


FOR  DEFENSE  ACQUISITION  CONTRACTS 


I .  Introduction 


General  Issue 

In  recent  years  the  United  States  military  services  have 
experienced  numerous  and  significant  cost  and  schedule 
overruns  on  their  major  weapon  systems  contracts.  For 
exan^le,  as  early  as  December  1990,  the  C-17  transport 
aircraft  was  experiencing  substantial  cost  overruns  (Smith, 
1990:36-39).  The  C-17  cost  and  schedule  overruns  have  not 
ed)ated.  In  fact,  "The  Defense  Department  revealed  Nov.  12 
[1993]  that  the  C-17  Globemaster  III  is  experiencing  a  new 
wave  of  schedule  setbacks  and  cost  overruns"  (Watkins, 
1993:2).  Also,  the  B-2  bomber  aircraft  has  had  huge  cost 
overruns  and  schedule  delays  (Velocci,  1993:29). 

Cost  and  schedule  problems  are  not  restricted  to  the  few 
high  profile  weapon  systems  identified  above.  In  fact,  "the 
average  cost  overrun  on  a  weapon  system  has  been  around  40 
percent"  (Gansler,  1989:4).  Severe  cost  overruns  and 
schedule  delays  may  be  due,  at  least  peurtly,  to  not 
expeditiously  determining  \dien  a  contractor  begins  to  have 
significant  difficulty  meeting  a  contract's  schedule  and/or 
cost  constraints.  An  analogy  can  be  made  to  a  small  hole  in 
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a  dike.  If  the  leak  is  realized  early  it  can  be  easily 
repaired.  However,  if  the  leak  is  left  unchecked  for  too 
long  the  leak  grows  severe  euid  it  becomes  very  difficult  to 
correct  the  problem. 

Obviously,  methods  that  can  allow  a  more  timely  means  of 
identifying  when  potentially  serious  cost  and  schedule 
overruns  begin  to  occur  would  be  beneficial.  This  would  aid 
weapon  system  program  managers  in  identifying  potential 
problems  eeurlier  when  there  €u:e  more  alternatives  available 
to  correct  the  situation.  One  approach  to  address  this  issue 
is  to  develop  quantitative  cost  and  schedule  benchmarks 
(benchmark  is  defined  as,  "a  point  of  reference  from  which 
measurements  of  any  sort  may  be  made"  (Webster's,  1971:203)) 
to  assist  decision  makers  in  identifying  potential  problems 
before  they  become  serious.  Cost  and  schedule  benchmarks  in 
the  context  of  this  thesis  should  not  be  construed  to  mean 
steuidcurds  of  excellence.  They  are  simply  descriptive 
indicators  of  past  cost  and  schedule  performance. 

Identifying  vdien  a  contract  is  "exceeding"  cost  and 
schedule  goals  is  not  as  simple  as  it  may  sound  since  so  many 
contracts  experience  some  degree  of  cost  and/or  schedule 
overrun.  The  challenge  is  determining  when  a  cost  or 
schedule  overrun  is  significant  enough  to  warrant  close 
management  scrutiny.  Due  to  limited  personnel  resources, 
program  managers  cannot  afford  to  investigate  every  cost  and 
schedule  deviation. 
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Deviations  from  a  cost  and  schedule  plan  are  known  as 
cost  €md  schedule  variances.  Cost  and  schedule  benchmarks 
represent  the  mean  cost  and  schedule  variances  on  past 
contracts.  These  benchmarks  can  be  used  to  evaluate  the 
overall  cost  and  schedule  status  of  ongoing  contracts.  Cost 
and  schedule  benchmarks,  along  with  their  associated  standard 
deviations,  can  assist  decision  makers  in  determining  what 
contracts  wcirrant  closer  management  attention  by  providing 
early  cost  auid  schedule  trend  information.  Then,  program 
managers  can  focus  their  limited  investigative  and  managerial 
resources  on  selected  contracts. 

Background 

In  an  attempt  to  gain  better  control  over  major  weapons 
progrctms,  the  government,  in  1967,  designed  standards  to 
encourage  contractors  to  focus  on  their  internal  cost  and 
schedule  management  control  systems.  These  standards,  called 
Cost /Schedule  Control  Systems  Criteria  (C/SCSC),  enable  the 
government  to  obtain  timely  and  auditable  data  from  the 
contractor  that  are  used  to  determine  contract  status. 

The  primary  vehicles  for  reporting  contract  data  is  the 
Cost  Performance  Report  (CPR)  euid  the  Cost  Schedule  Status 
Report  (C/SSR)  (AFMCP  173-5,  1987:2-1).  infoimation  from  the 
CPR  or  C/SSR  is  summarized  in  the  Defense  Acquisition 
Executive  Summary  (DAES)  report  and  then  it  is  entered  into  a 
centralized  database  called  the  DAES  database  (DODI  5000. 2M, 
1991:part  11).  This  database  gives  DoD  decision  makers 
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access  to  the  information  they  need  in  determining  the  status 
of  major  DoD  acquisition  programs. 

The  DAES  database,  which  contains  detailed  information 
on  over  500  military  contracts,  contains  the  quantitative 
information  necessary  to  create  cost  and  schedule  benchmarks. 
These  benchmarks  can  then  be  used  throughout  the  life  of  the 
military  contract  to  monitor  the  program  by  comparing  the 
current  status  of  the  contract  to  a  historical  reference.  If 
a  contract's  cost  and  schedule  status  varies  significantly 
from  the  established  benchmeirk,  vdien  considering  the 
associated  standard  deviation,  progrcun  managers  can  commit 
their  limited  resources  to  investigate  what  is  causing  this 
deviation. 

The  difference  between  benchmarks  and  benchmarking,  needs 
to  be  en^hasized.  Benchmarks,  as  used  in  this  study,  are 
indicators  of  cost  ajid  schedule  performance  based  on 
historical  data  which  can  be  used  to  gauge  future 
performance.  In  contrast,  "Benchmarking  is  the  continuous 
process  of  measuring  products,  services,  or  activities 
against  the  best  levels  of  performeuice  that  may  be  found 
either  inside  or  outside  the  organization”  (Homgren, 

1994:7).  Thus,  the  term  benchmarks  is  clearly  not  synonymous 
with  the  concept  of  benchmarking. 

Specific  Problem 

The  primary  purpose  of  this  research  is  to  calculate  and 
document  quantitative  cost  and  schedule  measures  of 
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performance,  based  on  historical  data,  at  various  stages  of 
militeury  contracts.  This  will  provide  program  managers  with 
benchmarks  with  which  to  determine  if  the  status  of  their 
contract  is  different  from  past  contracts. 

A  secondary  purpose  of  this  research  is  to  conpare  the 
benchmarks  produced  in  the  primary  stage  of  the  analysis 
across  categories  of  contracts  to  determine  if  there  is  a 
statistically  significcuit  difference.  Identifying 
statistically  significcuit  differences  between  cost  and/or 
schedule  vcurieuices  on  contracts  categorized  by  contract  type, 
program  phase,  and  military  service  might  lead  to 
improvements  in  contract  management  and  lower  cost  and 
schedule  variances. 

Investigative  Questions 

The  following  questions  were  used  to  direct  the  focus  of 
this  research: 

1.  What  general  circumstances  have  historically  caused 
cost  overruns  and  schedule  slippages? 

2.  What  are  the  historical  cost  and  schedule  variances 
for  the  12  categories  of  contracts  at  selected  stages? 

3.  Given  program  phase,  contract  type,  cuid  contract 
stage,  are  there  statistically  significant  cost  or  schedule 
differences  across  the  three  militeiry  services  (Air  Force, 
Array,  and  Navy)? 

4.  Given  contract  type,  managing  service,  and  contract 
stage,  aure  there  statistically  significant  cost  or  schedule 
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differences  between  the  two  program  phases  (development  and 
production)? 

5.  Given  program  phase,  managing  service,  euid  contract 
stage,  are  there  statistically  significant  cost  or  schedule 
differences  between  the  two  contract  types  (cost  reimbursable 
and  fixed  price)? 

6.  How  can  cost  and  schedule  benchmarks  be  used  to 
assist  program  managers  in  determining  the  status  of  on-going 
contracts? 

The  Primary  Hypotheses  Tested 

1.  Tested  a  null  hypothesis  (Ho)  that  given  program 
phase,  contract  type,  and  contract  stage,  there  is  no 
statistically  significant  difference  among  the  cost  or 
schedule  variances  of  contracts  managed  by  the  Air  Force, 
Army,  and  Navy. 

2.  Tested  a  null  hypothesis  (Ho)  that  given  contract 
type/  nanaging  service,  and  contract  stage  there  is  no 
statistically  significant  difference  among  the  cost  or 
schedule  variauices  between  the  two  program  phases 
(development  and  production). 

3.  Tested  a  null  hypothesis  (Hq)  that  given  program 
phase,  contract  type,  and  contract  stage  there  is  no 
statistically  significant  difference  eunong  the  cost  or 
schedule  variances  between  the  two  contract  types  (fixed 
price  and  cost  reimbursable). 


6 


Scope/Limitations /Assumptions 

This  research  is  confined  to  the  military  contracts 
contained  in  the  DAES  database.  Contracts  which  are  excluded 
from  the  database  include  contracts  classified  as  "black” 
programs  for  national  security  reasons,  contracts  conqpleted 
prior  to  1972,  and  contracts  not  meeting  est£d}lished 
reporting  thresholds,  in  addition,  contracts  in  the  full 
DAES  datcibase  that  had  missing  data  considered  essential  to 
this  analysis  were  eliminated.  The  refinements  made  to  the 
datsLbase  in  order  to  tailor  it  to  this  study  are  discussed  in 
detail  in  the  methodology  section. 

The  "refined"  DAES  datcibase  only  contains  contracts  that 
have  con^rehensive  cost  and  schedule  data.  Thus,  a  judgment 
sanqple  was  used  for  statistical  analysis.  Because  the  DAES 
database  is  a  non-random  sao^ling  of  the  population  of  all 
military  contracts,  caution  should  be  used  when  making 
inferences  concerning  the  population.  As  a  result,  the 
inferential  statistics  produced  in  this  study  can  only  be 
relied)ly  applied  to  those  contracts  with  the  same  attributes 
as  those  currently  included  in  the  DAES  database. 

The  calculated  benchmarks  €u:e  limited  by  the  accuracy  of 
the  information  in  the  DAES  database.  Inaccurate  data  can 
result  from  human  error  in  either  con^leting  the  cost 
performance  report  or  entering  the  data  into  the  database. 
Also,  in  rare  situations  it  is  possible  that  the  data 
reported  on  the  CPR  were  erroneous.  Given  the  size  of  the 
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sample  in  the  study  (462  contracts),  errors  in  the  data 
should  not  have  a  material  effect  on  the  results  of  the 
analysis . 

Although  cost  overruns  and  schedule  problems  are  often 
rooted  in  the  contractor's  performance,  this  thesis  should 
not  be  interpreted  as  faulting  the  contractor  for  all  cost 
overruns  and  schedule  slippages.  Due  to  time  and  resource 
limitations  it  was  not  possible  to  determine  all  of  the 
factors  that  might  have  adversely  injected  every  specific 
program's  cost  and  schedule  status.  General  situations  that 
adversely  impact  cost  and  schedule  are  discussed  in  the 
literature  review  section.  The  program  manager /decision 
maker  must  determine  what,  if  any,  other  factors  are 
adversely  inpacting  the  progreun  before  focusing  on  the 
contractor  as  the  cause  of  the  overruns. 

Thesis  Overview 

This  thesis  utilizes  the  DAES  datcd^ase  to  calculate 
benchmarks  for  determining  the  cost  and  schedule  status  of 
current  contracts  as  conpared  to  past  contracts. 

The  Literature  Review,  Chapter  Two,  summarizes  past 
efforts  in  the  €u:eas  of: 

1.  The  factors  that  adversely  inpact  cost  and  schedule 

2.  How  Cost/Schedule  Control  Systems  Criteria  apply  to 
the  collection  and  documentation  of  contract  data 
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3.  The  informational  structure  of  the  DAES  database  and 
how  that  information  might  be  used  to  determine  contract 
status 

4.  What  the  level  of  need  is  for  additional  tools  to 
evaluate  contract  status 

The  Methodology  section/  Chapter  Three,  presents  how  the 
data  were  analyzed  and  what  specific  steps  were  followed  in 
euiswering  the  investigative  questions  and  testing  the 
hypotheses . 

The  Analysis  section.  Chapter  Four,  presents  the  . 
quantitative  results  of  the  statistical  analysis.  The 
hypotheses  were  tested  using  the  analysis  of  variance  (ANOVA) 
approach  and  the  significant  results  were  discuss  and 
summarized. 

Conclusions  based  on  the  analysis  are  drawn  in  Chapter 
Five.  In  addition,  recommendations  on  how  the  results  of 
this  study  could  be  utilized  to  iii^)rove  contract  management 
cu:e  included.  Basically,  the  cost  and  schedule  benchmarks 
could  be  used  to  monitor  contract  status  throughout  the  life 
of  a  military  contract. 
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II 


Literature  Review 


The  review  of  the  existing  literature  identifies  and 
discusses  pertinent  curticles,  theses,  reports  and  other 
published  works  related  to  four  areas: 

1.  The  factors  that  adversely  impact  cost  and  schedule 
perfonucuice 

2.  How  C/SCSC  applies  to  collecting  euid  documenting 
performance  data 

3.  How  the  DAES  database  is  structured  and  how  it  can 
be  used  to  determine  how  a  contract  is  progressing 

4.  What  level  of  need  exists  for  additional  tools  to 
evaluate  contract  status. 

Overview  of  Literature  Review 

Ten  published  works  were  reviewed.  Table  1  on  the 
following  page  contains  a  summary  of  these  published  works. 
The  first  one  (Woodward,  1983)  was  chosen  because  it 
thoroughly  covered  the  risk  and  uncertainty  factors  that 
influence  cost  and  schedule  performcuice.  Several 
authoritative  references  on  C/SCSC  were  written  by  Messrs. 
Fleming  (1988),  Christensen  (1993),  and  Abba  (1986).  These 
readings  were  selected  because  of  their  thorough  explanation 
of  the  C/SCSC  process.  Information  pertaining  to  the  DAES 
database  was  gathered  from  Christensen's  articles.  His 
writings  explained  the  nature  of  the  DAES  datcdsase  and  how 
this  database  is  useful  for  performance  measurement 
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calculations.  The  Beach  Report  (1990)  highlights  the  need 
for  more  accurate  eind  timely  cost  and  schedule  status 
information.  Mr.  Beach  clearly  points  out  the  consequence 
(the  cancellation  of  a  major  program)  resulting  from 
inadequate  cost  and  schedule  monitoring.  Lastly,  two  AFIT 
theses  detailing  some  alternative  cost  and  schedule 
performance  indicators  were  examined.  These  were  selected 
because  they  were  pertinent  to  the  topic  of  cost  and  schedule 
perfonBcuice  benchmarks. 


Table  1 

Summary  Of  Literature  Review 


lAOTHOR/YEAR  TOPIC  CONCLUSIONS 

Woodward/ 1983 

Risk  and 

Uncertainty 

Numerous  causes  of 
cost  overruns 

Fleiiiina/1988 

C/SCSC 

N/A 

Christensen/ 1993 

C/SCSC 

N/A 

AFMCP  173-5/1987 

C/SCSC 

N/A 

Christensen/1992 

Control  &  Analysis 
of  Performance 
Measurement  Data 

N/A 

Christensen / 1 9  9  3 

Cost  Overruns 

Use  performance 
measurement  data 
for  timely 
identification  of 
overruns 

Abba/1986 

C/SCSC 

Take  corrective 
actions  for  cost 
euid  schedule 
deviations  early 

Knepp  & 

Stroble/1993 

S-curves  for  cost 
control 

S-curves  couldn't 
be  used 

Terry  & 

Vanderburqh/ 1993 

EAC  Indices 

SCI -based  EAC  is 
accurate 

Beach/ 1990 

A-12  Aircraft 

Need  to  recognize 
cost  overruns 
earlier 
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General  Factors  Affecting  Cost  Overruns  and  Schedule 
Slippages 

There  are  meuiy  factors  that  affect  the  cost  and  schedule 
status  of  defense  acquisition  systems.  Although 
inefficiencies  in  the  contractor's  operation  can  cause  cost 
overruns  and  schedule  slippages,  other  factors  often  play  a 
part. 

Geuisler  points  out  that  the  procurement  system 
"encourages  a  great  deal  of  "optimism"  in  bidding  and  in 
budgeting"  (Gansler,  1989:177).  Because  the  contract  is 
often  awarded  to  the  lowest  bidder,  it  behooves  a  contractor 
to  bid  the  lowest  possible  amount  for  the  contract.  This  is 
not  to  imply  the  contractor  is  purposely  understating  the 
actual  cost  of  fulfilling  the  contract.  Indeed,  the  whole 
concept  of  risk  analysis  and  probability  distributions 
implies  there  are  a  great  number  of  possibilities  over  what 
the  contract  will  really  cost.  Soma  of  these  probabilities 
are  small  but  are  non-the-less  possibilities. 

Another  reason  for  cost  overruns  is  a  greater  government 
priority  on  minimizing  development  time  regardless  of  the 
cost  consequences  (Peck,  1962:438).  This  is  not  necessarily 
always  bad.  When  the  United  States  is  on  the  verge  of  war  it 
is  important  to  get  the  weapon  systems  fielded  as  soon  as 
possible  in  order  to  increase  the  advantage  over  the  enemy 
and  reduce  casualties  by  minimizing  the  duration  of  the 
conflict.  In  this  situation  cost  ramifications  become 
secondary.  Expediting  development  time  at  the  expense  of 
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controlling  costs  becomes  poor  management  when  there  are  no 
clear  cut  national  priorities  for  doing  so  except  just  a 
desire  to  get  the  system  on-line  sooner. 


Woodward  Thesis 

Factors  that  Woodward  noted  in  his  1983  thesis,  ^ 


1.  Not  adequately  budgeting  for  risk  and  uncertainty 
(particulairly  during  the  relatively  high  risk  R&D  phase) 

2.  unexpected  high  inflation 

3.  Supply  and  demand  factors 

4.  Poor  resource  allocation 


5.  Managerial  inefficiency 

6 .  Technological  uncertainty 

7.  System  requirements  uncertainty 

8.  Less  than  perfect  cost  estimating  techniques 

9.  The  budget  process  involving  risk  in  congressional 
appropriations 

The  adverse  im^ct  that  uncertainty  in  the  congressional 
appropriations  process  can  have  on  a  major  weapons  program 
can  be  readily  seen  in  the  acquisition  of  the  Air  Force's 
next  generation  air  superiority  fighter.  A  recent  $163 
million  congressional  cut  in  the  development  phase  of  the 
F-22  could  "translate  into  at  least  $400  million  in  added 


development  costs  for  the  F-22  and  stretch-out  development  by 
five  months"  (Cole,  1994 :A-3).  The  author  went  on  to  state. 
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Every  time  F-22  funding  for  a  given  year  is  cut,  the 
total  cost  of  developing  the  plane  actually  escalates. 
When  the  funding  is  cut  in  the  short  term,  development 
must  be  slowed,  meaning  work  with  lots  of  fixed<overhead 
costs  is  stretched  out  over  more  time.  The  higher  price 
invariedsly  gives  political  opponents  more  ammunition  for 
cutting  the  program  further.  Gen.  Merrill  McPeak,  Chief 
of  Staff  of  the  Air  Force,  warned  in  an  interview  that 
the  F-22  was  in  danger  of  being  pushed  into  'the  same 
old  death  spiral  we  always  get  into  on  these  programs . ' 

Woodward  concluded  his  research  by  stating,  "Further 

research  should  be  done  in  the  management  control  of  risk  and 

uncertainty,  as  well  as  cost  and  schedule  variances  within 

government  programs"  (italics  added)  (Woodward,  1983:108). 

As  previously  stated,  calculating  and  analyzing  cost  and 

schedule  variances  with  the  objective  of  designing  cost  and 

schedule  benchmarks  is  the  objective  of  this  thesis. 

It  must  be  eiq)hasized  that  there  £ire  many  reasons  why 

program  costs  and  schedules  can  be  adversely  affected.  The 

program  manager  needs  to  carefully  analyze  the  various 

influences  on  the  program  before  concluding  the  root  cause 

lies  in  the  contractor's  performance.  Fortunately,  major 

defense  programs  are  governed  by  Cost /Schedule  Control 

Systems  Criteria  designed  to  facilitate  identifying  programs 

experiencing  difficulty  meeting  cost  and  schedule  goals. 


Cost/Schedule  Control  Systems  Criteria 

The  Cost/Schedule  Control  Systems  Criteria  were 
developed  in  1967  to  address  fundamental  deficiencies  in  the 
management  of  defense  acquisition  programs.  Prior  to 
implementation  of  C/SCSC  contract  performance  data  were  often 
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incomplete,  inconsistent,  and  inaccurate.  The  35  criteria, 
or  standards,  contained  in  C/SCSC  were  designed  to 
standcurdize  performance  data  :(:eporting  and  provide  decision 
makers  with  thorough,  accurate  and  reliadjle  information. 

When  a  contractor  configures  their  management  control 
systems  in  accordance  with  the  35  criteria,  they  are 
certified  as  C/SCSC  compliant  by  the  government.  Thereafter, 
the  government  places  high  confidence  in  the  data  the 
contractor  reports  on  contracts  under  their  control.  It  must 
be  enphasized  that  contractors  are  not  required  to  design  any 
one  specific  system  to  become  certified.  However,  they  must 
ensure  their  systems  are  C/SCSC  compliant. 

The  government  realizes  that  no  one  management  control 
system  is  perfect  for  every  contractor  given  the  diverse 
nature  of  DoD  acquisitions.  As  a  result,  contractors  have 
wide  latitude  in  designing  management  control  systems  to  meet 
their  unique  needs  provided  the  system  satisfies  the  35 
criteria  needed  to  ensure  accurate  and  relicUale  cost  and 
schedule  information. 

Contractors  use  the  Cost  Performance  Report  (CPR)  or  the 
Cost/Schedule  Status  Report  (C/SSR)  to  report  cost  and 
schedule  information  on  their  contracts.  According  to 
Department  of  Defense  Cost /Schedule  Control  Systems  Criteria 
Joint  Impi emantation  Guide  .  the  CPR  contains  data  that  must: 

1.  relate  time-phased  budgets  to  specific  contract 

tasks  and/or  statements  of  work; 

2.  indicate  work  progress; 


15 


3.  properly  relate  cost,  schedule  and  technical 
accomplishment ; 

4.  be  valid,  timely,  and  auditable; 

5.  supply  DoD  managers  with  information  at  a  practical 
level  of  summarization;  and 

6.  be  derived  from  the  same  internal  management  control 
systems  used  by  the  contractor  to  manage  the  contract. 
(APMCP  173-5,  1987:1-1) 

Key  information  from  the  CPR  is  entered  into  the  DAES  report 
and  then  into  a  summary  datcUPase  to  allow  decision  makers 
rapid  access  to  determine  the  cost  and  schedule  performance 
status  of  current  and/or  con^leted  major  defense  contracts. 

Defense  Acquisition  Executive  Summary  (DABS)  Database 
The  Defense  Acquisition  Executive  Summary  (DAES) 
database  is  a  compilation  of  cost  euid  performance  data  on 
over  500  Department  of  Defense  (DoD)  contracts.  The 
information  in  the  dataUsase  covers  most  of  the  major  weapon 
systems  contracted  for  development  or  production  by  the  DoD 
since  1972. 

Data  from  the  CPRs  and  C/SSRs  are  summarized  and  sent, 
on  a  quarterly  basis,  to  the  Office  of  the  Undersecretary  of 
Defense  for  Acquisition  (OUSD(A))  (Christensen,  1993)  (DODI 
5000. 2M,  1991tpart  11).  This  summary  is  called  the  Defense 
Acquisition  Executive  Summauy  (DAES).  Data  from  this  summary 
are  entered  into  the  database.  Information  in  the  database 
is  proprietary,  therefore,  information  that  could  be  used  to 
identify  the  contractor  or  a  specific  weapon  system  will  not 
appear  in  this  thesis.  Specific  responsibilities  for  all 
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participants  in  the  reporting  process  are  detailed  in  DoD 
Manual  5000. 2M,  part  11. 

The  database  is  quite  detailed;  it  contains  over  50 
fields  of  data.  The  current  database  has  over  7,000  lines  of 
data  representing  completed  eind  on-going  contracts  from  the 
eeirly  1970 's  to  date. 

Some  of  the  fields  in  the  database  used  in  determining 
the  status  of  cost  overruns  and  schedule  problems  cure: 

(note:  formal  definitions  of  these  terms  cure  in  Appendix  A) 

1.  Cumulative  Budgeted  Cost  of  Work  Scheduled  (BCWS). 
This  figure  is  also  known  as  planned  value.  It  is  the  value 
of  the  work  the  contractor  had  planned  to  complete  by  the 
reporting  date.  If  the  contractor  had  completed  an  amount  of 
work  equal  to  this  figure  he  would  be  on  schedule  (i.e.  zero 
schedule  variance) 

2.  Cumulative  Budgeted  Cost  of  Work  Performed  (BCWP). 
This  figure  is  also  known  as  earned  value.  This  amount 
represents  the  amount  of  the  contract  the  contractor  has 
actually  earned.  If  BCWP  does  not  equal  BCWS  then  the 
contractor  did  not  complete  the  exact  amount  of  work  he  had 
planned  to  complete;  he  would  have  completed  more  work  or 
less  work  than  what  was  planned.  Differences  between  planned 
auid  actual  work  completed  is  known  as  a  schedule  variance 
(SV)  and  is  calculated  as: 

Schedule  Varieuice  (SV)  »  BCWP  -  BCWS  (1) 
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significant  schedule  variances,  either  positive  or  negative, 
are  cause  for  concern  since  the  work  would  not  be  progressing 
according  to  the  plan. 

3.  Cumulative  Actual  Cost  of  Work  Performed  (ACWP). 

This  eunount  represents  the  actual  cost  of  the  work  performed 
as  of  the  time  the  CPR  was  submitted.  If  ACWP  does  not  equal 
BCWP  then  the  contractor  spent  more  or  less  money  than  what 
was  plcuuied  for  the  amount  of  work  actually  performed.  A 
difference  between  planned  costs  etnd  actual  costs  is  known  as 
a  cost  varicuice  (CV)  and  is  calculated  as: 

Cost  Variance  (CV)  =  BCWP  -  ACWP  (2) 

This  variance  will  be  positive  if  the  contractor  spent  less 
than  budgeted  for  the  actual  work  performed  or  negative  if  he 
spent  more  than  budgeted  for  the  actual  work  performed. 
Significemt  cost  variances,  either  positive  or  negative,  are 
cause  for  concern  since  costs  are  not  close  to  what  was 
anticipated  when  the  budget  was  drafted. 

The  equations  for  determining  cost  euid  schedule 
variances  were  taken  from  Control  and  Analysis  of  Performance 
Measurement  Data  (Christensen,  1992:20). 

Christensen  Article 

David  S.  Christensen  published  an  euialysis  of  cost 
overruns  on  DoD  acquisition  contracts.  In  his  discussion,  he 
stresses  that  "timely  management  attention  to  adverse  cost 
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varieuices  can  reduce  them.  .  .the  problem  has  been  a  failure 

to  use  performance  measurement  data  proactively” 

(Christensen,  1993:44-45).  This  research  addresses 

Christensen's  concerns  regarding  the  timely  use  of 

performance  measurement  data  by  providing  cost  and  schedule 

benchmarks  ^ich  program  managers  can  use  to  assess  the 

severity  of  potential  problems  early  in  the  program.  This 

may  allow  additional  time  to  take  appropriate  intervening 

action.  Christensen  concludes  by  stating: 

Without  more  realistic  estimates  [of  cost  and/or 
schedule  variances],  senior  management  may  be  lulled 
into  a  false  sense  of  security  about  their  programs  and 
fail  to  take  appropriate  action  to  correct  problems. 
(Christensen,  1993:47) 

Abba's  C/SCSC  White  Paper 

Mr.  Wayne  Abba,  of  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  prefaced  the  Cost/Schedule  Control 
Systems  Criteria  Joint  Implementation  Guide,  by  addressing 
the  importance  of  C/SCSC.  He  advocates  using  C/SCSC  for 
collecting  and  tracking  both  cost  and  schedule  data  to  assess 
contractors'  performance.  He  emphasizes,  "Real  improvements 
in  contract  management  can  be  achieved  by  top-level  attention 
to  developing  and  using  good  cost  and  schedule  mauiagement 
control  systems  and  by  taking  timely  corrective  action  when  a 
problem  is  identified”  (AFMCP  173-5,  1986:viii). 
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Knepp  and  Stroble  Thesis 

Richard  Knepp  cuid  Michael  Stroble  in  their  1993  thesis 
Development  of  Standardized  S  Curves  for  the  Evaluation  of 
Major  Department  of  Defense  Purchases  investigated  using  S- 
curves  to  describe  how  costs  react  over  time  for  typical 
major  weapons  programs  (Knepp  and  Stroble,  1993).  They  also 
used  the  DAES  database  in  their  analysis.  They  used  317 
contracts  in  their  study.  They  had  hoped  to  provide  program 
managers  with  a  cost-based  S-curve  created  from  historical 
data.  Program  managers  could  then  use  this  S-curve  to  help 
gauge  the  status  of  their  contracts. 

Knepp  £uid  Stroble  theorized  that  weapon  system  contracts 
would  exhibit  a  familiar  trend  in  costs  as  the  contract  is 
completed.  Basically,  they  hypothesized  costs  are  slow  to 
increase  in  the  early  stages  of  a  contract  and  then  increase 
fairly  rapidly  in  the  middle  stages  of  the  contract  before 
tailing  off  toweurd  the  end  of  the  contract;  thus  exhibiting 
an  "S"  shape.  Unfortunately,  they  discovered  the  pattern  of 
costs  with  respect  to  percent  complete  was  often  more  linear 
that  S-shaped.  Howsver,  even  though  a  distinctive  S-curve 
was  not  always  evident,  the  reference  line  odght  have  still 
proved  useful  to  program  managers  to  gauge  the  status  of 
their  current  contracts  except  for  a  key  point,  the  spread  of 
the  aggregated  cost  veuriance. 

The  aggregated  cost  variance  in  the  study  was  spread 
over  a  large  area  (i.e.  the  standard  deviation  was  very 
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Icucge).  For  example,  given  a  contract  that  was  50%  complete, 
the  expected  cost,  plus  or  minus  one  standard  deviation,  was 
predicted  to  be  between  approximately  42%  and  72%  of  total 
contract  cost.  Basically,  this  is  indicating  that,  based  on 
historical  data,  a  contract  is  "in  the  window"  if  cumulative 
costs  were  between  42%  and  72%  of  total  contract  costs  at  the 
50%  coiq)letion  point. 

It  appears  that  due  to  the  Icurge  size  of  the  "window,  ’’ 
progrcun  managers  would  have  difficulty  using  this  information 
to  determine  how  their  contracts  are  performing  relative  to  a 
historical  reference.  Unless  the  cumulative  costs  greatly 
exceeded  the  historical  norm,  decision  makers  would  have  to 
conclude  the  contracts  are  within  cost  boundaries. 

Terry  and  Vanderburgh  Thesis 

In  1993,  Mark  Terry  and  Mary  Vanderburgh  performed  an 
analysis  of  estimate  at  completion  (EAC)  models  utilizing  the 
DAES  datcUaase.  in  their  thesis.  An  Analysis  of  BstimatiP  at- 
Completion  Models  Utilizing  the  Defense  Acquisition  Executive 
snimnarY  Database  they  focused  on  using  various  indices  to 
estimate  a  floor  euid  ceiling  for  the  final  cost  of  a 
contract.  The  three  indices  they  used  in  their  analysis  were 
the  Cost  Performance  index  (CPI),  the  Schedule  Cost  index 
(SCI),  and  the  Schedule  Performance  Index  (SPI).  By 
identifying  a  floor  auid  ceiling  for  the  estimated  costs  of  a 
contract,  they  might  be  able  to  predict  realistic  cost  bounds 
for  a  given  contract. 
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Terry  and  Vanderburgh  used  321  contracts  from  the  DAES 
database  in  their  analysis.  They  concluded  that  the  CPI- 
based  estimate  at  coiq>letion  (EAC)  represents  a  valid  floor 
for  a  contract's  final  cost  estimate.  Although  they  expected 
the  SCI-based  EAC  would  be  a  valid  cost  ceiling,  they 
discovered  the  actual  cost  at  completion  (CAC)  exceeded  the 
SCI-based  EAC.  However,  of  the  EACs  they  analyzed,  the  SCI- 
based  EAC  was  the  most  accurate  predictor  of  final  contract 
cost. 

In  summary,  Terry  and  Vanderburgh  analyzed  the 
appropriateness  of  using  various  performance  indices  to 
predict  contract  costs  at  con^letion  and  had  positive 
results.  They  recommended  that  "a  more  detailed  statistical 
analysis  of  the  DAES  dated)ase  would  be  very  useful  for  future 
research."  Analyzing  the  DAES  datcd>ase  with  the  goal  of 
producing  descriptive  statistical  cost  and  schedule 
benchmarks  is  a  logical  extension  of  their  work. 

Beach  Report 

Navy  Inquiry  Officer,  Chester  Paul  Beach  Jr.,  issued  a 
memoreuidum  on  the  status  of  the  A-12  aircraft  program  for  the 
Secreteu:y  of  the  Navy  on  28  Nov  1990  (Beach,  1990).  Forty 
days  later.  Defense  Secretary  Cheney  formally  canceled  the  A- 
12  program. 

Beach's  report  highlighted  many  problems  with  the  A-12 
progreun,  including  the  fact  that  the  progrcua  manager's  office 
did  not  formally  recognize  t!iat  the  A-12  contract  was  in 
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trouble  until  it  was  in  such  serious  difficulty  that  the 
Secretairy  of  Defense  had  to  step  in  to  terminate  it.  The 
problems  of  the  A-12  program  have  helped  focus  the  attention 
of  the  defense  contracting  community  on  the  need  for  tools 
that  program  managers  can  use  to  assess  the  viability  of 
their  weapon  or  material  programs.  The  benchmarks  this 
thesis  calculates  are  intended  to  be  one  of  these  assessment 
tools.  Beach's  conclusions  concerning  the  A-12  aircraft 
program  validate  the  need  for  cost  and  schedule  benchmarks. 

Investigative  Question  Answered 

There  are  numerous  reasons  why  cost  overruns  and 
schedule  slippages  occur  (investigative  question  #1).  These 
reasons  include:  poor  contractor  perform2uice,  not  properly 
accounting  for  risk  and  uncertainty  when  budgeting  funds  for 
the  contract,  not  properly  managing  the  contract  to  discover 
vdien  cost  and  schedule  problems  are  occurring,  and 
congressional  budget  cuts  in  the  middle  of  the  contract  that 
cause  unexpected  euid  short  notice  schedule  slips. 

Given  that  cost  and  schedule  problems  are  occurring  in  a 
large  percentage  of  contracts,  there  is  a  need  for  a  means  of 
identifying  cost  and  schedule  deviations  at  the  earliest 
possible  time.  Cost  and  schedule  benchmarks  should  prove 
useful  to  the  program  manager  by  providing  early  indications 
of  when  a  contract  is  exceeding  historical  average  cost  cUid 
schedule  variances. 
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Conclusion  and  Overview 

Although  the  C/SCSC  management  control  system  is 
designed  to  minimize  cost  and  schedule  overruns,  problems  in 
these  areas  remain.  Studies  have  explored  ways  to  address 
cost  and  schedule  issues  with  limited  success.  The  next 
chapter  presents  the  specific  methodology  used  in  the 
analysis . 
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III.  Methodology 


Introduction 

This  section  specifies  the  methodology  used  in  this 
analysis.  A  refined  DAES  database  of  290  military  contracts 
was  used  to  calculate  benchmarks  for  cost  and  schedule 
varieuices  (in  percent)  of  contracts.  This  study  employs 
secondary  data.  It  uses  quantitative  analysis  producing 
inferential  statistics.  In  addition.  Analysis  of  Variance 
(AMOVA)  is  used  to  analyze  the  data. 

Overview  of  Benchmark  Calculations 

Benchmairks  are  sisqply  the  mean  cost  and  schedule 
variances  (in  percent)  for  designated  categories.  The  cost 
and  schedule  variances  calculated  from  the  DAES  database  were 
grouped  into  48  categories.  These  categories  were  derived 
from  the  three  military  services  under  analysis  (Air  Force, 
Army  or  Navy),  the  two  possible  phases  of  a  contract 
(development  or  production),  the  two  general  types  of 
contracts  (cost  reimbursable  or  fixed  price),  and  the  four 
stages  of  a  contract  (0-25%  coii^>lete,  25-50%  complete,  50-75% 
complete,  or  75-100%  conqplete).  This  breakout  resulted  in  48 
cost  veu:i£uice  and  48  schedule  variance  benchmarks 
(3*2*2*4*48). 

Statement  of  Hypotheses 

The  six  hypotheses  are: 
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1.  Given  the  contract  phase,  contract  type  euid  contract 
stage,  there  is  no  statistically  signific2nit  difference  in 
the  cost  variance  percentages  of  the  three  militairy  services. 
The  null  and  alternative  hypotheses  are: 

Ho*  ^^M'Air  Force  “  CV%|lAxiiiy  =  CV%^ifavy 
Ha*  not  all  three  means  are  equal 

2.  Given  the  contract  phase,  contract  type  and  contract 
stage,  there  is  no  statistically  significant  difference  in 
the  schedule  varicuice  percentages  of  the  three  military 
services.  The  null  auid  alternative  hypotheses  are: 

Ho*  SV%p,j^r  Force  ~  SV%fiAnny  ~  SV%JlNavy 
Ha*  not  all  three  mecins  cure  equal 

3.  Given  the  military  service,  contract  type  and 
contract  stage,  there  is  no  statistically  significant 
difference  in  the  cost  variance  percentages  of  the  two 
contract  phases.  The  null  and  alternative  hypotheses  are: 

Ho*  CV%fADevelopment  =  CV%(iproduction 
Ha*  CV%p,Dovelopinent  ^  CV%^.production 

4.  Given  the  military  service,  contract  type  and 
contract  stage,  there  is  no  statistically  significant 
difference  in  the  schedule  variemce  percentages  of  the  two 
contract  phases.  The  null  euid  alternative  hypotheses  are: 

Ho*  HV%^.]>«v«lopaient  ~  SV%p,production 
Ha*  HV%|ADeyeXoiiiient  ^  SV%p.production 

5.  Given  the  military  service,  contract  phase  and 
contract  stage,  there  is  no  statistically  significant 
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difference  in  the  cost  variance  percentages  of  the  two 
contract  types.  The  null  and  alternative  hypotheses  are: 

Ho*  CV%^,co8t  Reimbursable  =  CV%[lpixed  Price 
Ha*  CV%^coat  Reimbursable  ^  ^^^Fixed  Price 

6.  Given  the  military  service,  contract  phase  and 
contract  stage,  there  is  no  statistically  significant 
difference  in  the  schedule  variance  percentages  of  the  two 
contract  types.  The  null  and  alternative  hypotheses  are: 

Ho*  SV%ficost  Reimbursable  =  SV%|ipixed  Price 
Ha*  SV%Jlcoat  Reimbursable  ^  SV%^yixed  Price 

Relevance  of  Hypotheses  Testing 

A  benchmark  may  be  affected  by  the  following  categories: 
program  phase,  contract  type,  and  branch  of  military  service. 
Hatching  the  characteristics  of  an  ongoing  contract  with  the 
benchmarks  would  be  appropriate  if  the  benchmark  does  in  fact 
depend  on  these  categories.  Therefore,  hypothesis  testing  is 
necesscury  to  determine  if  statistically  significant 
differences  between  benchmcurks  truly  exist. 

The  DABS  Database 

The  DAES  datcUOase  is  the  principal  cost  performance 
reporting  database  for  the  three  military  services  and  is 
coiq>rised  of  quarterly  cost  and  schedule  reports  which  are 
sutmitted  by  the  program  managers  (Beach,  1990:2). 

Coiq>emies  with  large  military  contracts  must  comply  with 
the  government's  Cost /Schedule  Control  Systems  Criteria 
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(C/SCSC)  and  normally  report  detailed  cost  cuid  schedule 
information  to  the  program  managers  throughout  the  life  of 
the  contract.  The  Cost  Performance  Report  (CPR)  or  the 
Cost /Schedule  Status  Report  (C/SSR)  are  used  for  reporting 
this  data.  Program  managers  then  summarize  the  information 
the  contractor  submitted  for  insertion  into  the  DAES  database 
via  the  DAES  report. 

Population  &  Sample 

The  population  is  DoO  military  contracts  contained  in 
the  Defense  Acquisition  Executive  Summary  (DAES)  database. 

For  this  analysis,  a  judgment  sample  was  extracted  from  the 
DAES  datcLbase. 

The  database  was  refined  by  eliminating  those  contracts 
which  could  not  be  specifically  categorized  into  either  the 
production  or  development  phase,  or  into  cost  reimbursable  or 
fixed  price  type  of  contracts.  Contracts  with  blank  fields 
identifying  the  phase  or  type  of  contract  were  eliminated. 
Individual  contracts  which  listed  both  development  and 
production  and/or  cost  reimbursable  and  fixed  price  were  not 
considered  since  they  did  not  conform  to  the  collectively 
exhaustive  and  mutually  exclusive  categories  in  the  study. 

Additionally,  contracts  that  had  missing  data  or  clearly 
incorrect  data  due  to  input  errors  in  fields  considered 
essential  to  this  study  (ACWP,  BCWP  and  BCWS)  were 
eliminated.  Thus,  the  sample  consists  of  those  contracts  for 
vdiich  there  cure  valid  and  complete  data.  "Complete" 
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contracts  are  defined  as  those  contracts  having  entries  in 
these  fields:  Militciry  Service,  Contract  Phase,  Contract 
Type,  Final  Budget  at  Completion  (FBAC),  BCWS,  BCWP  cuid  ACWP. 

Data  Collection  Instrument 

The  data  for  the  OAES  datcibase  were  collected  through 
detailed  Cost  Performance  Reports  (CPR)  auid  the  Cost /Schedule 
Status  Report  (C/SSR)  and  are  maintained  in  the  DAES  database 
as  historical  data  (AFMCP  173-5,  1987:2-1)  (DODI  5000. 2M, 
1991:16-1  to  16-7  2uid  20-1  to  20-9).  These  documents  lay  out 
the  specific  criteria  with  which  contractors  must  con^ly  on 
military  resecirch,  development,  or  test  and  evaluation 
contracts  over  $60  million  and  any  procurement  contract  over 
$250  million  in  FY90  constant  dollars.  Due  to  the  detailed 
C/SCSC  contractor  performance  reporting  standards  and  the 
presence  of  military  representatives  at  the  contracting 
sites,  the  data  in  the  DAES  datcUsase  are  assumed  valid  for 
this  study. 

Data  Collection  Plan 

The  data  used  in  this  study  had  previously  been 
collected  by  a  third  source.  The  DAES  database  was  provided 
to  the  researchers  by  David  S.  Christensen  of  the  Air  Force 
Institute  of  Technology.  Mr.  Christensen  originally  received 
the  database  from  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition.  The  cost  and  schedule  data  are 
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ratio  data  with  zero  dollars  and  zero  time  being  the 
£d)SOlutes . 

The  data  were  appropriately  categorized  for  this 
research.  Some  examples  of  categories  are:  contract  number 
(a  numeric  identifier),  branch  of  service  (Air  Force,  Army  or 
Navy),  program  phase  (development  or  production),  type  of 
contract  (coat  reimburscdsle  or  fixed  price),  budgeted  cost  of 
work  scheduled  (BCWS),  budgeted  cost  of  work  performed 
(BCWP),  and  actual  cost  of  work  performed  (ACWP). 

Data  Preparation 

There  were  5  steps  taken  to  prepare  the  data  for 
analysis. 

1.  Each  contract  was  separated  into  four  stages  prior 
to  calculating  the  actual  cost  and  schedule  variances  in 
percent.  These  stages  were  selected  to  provide  program 
managers  with  an  appropriate  level  of  detailed  information. 
The  variances  were  categorized  using  the  following 
definitions  where  Percent  Coiqplete  (PC)  is  defined  as 
Budgeted  Cost  of  Work  Performed  (BCWP)  divided  by  Final 
Budget  at  Ccxnpletion  (FBAC)  (note:  Final  BAC  was  used  in  lieu 
of  BAC  since  BAC  will  fluctuate  throughout  the  life  of  a 
contract  as  modifications  to  the  contract  produce  changes  in 
the  cost  of  the  contract — usin'g  FBAC  more  realistically 
reflects  the  actual  completion  status  of  the  contract): 

Percent  Complete  (PC)  =  BCWP  +  FBAC  (3) 
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stage  1  =  contracts  between  0%  and  25%  complete 

Stage  2  =  contracts  between  25.1%  and  50%  complete 

Stage  3  =  contracts  between  50.1%  and  75%  complete 

Stage  4  =  contracts  between  75.1%  and  100%  complete. 

2.  The  cost  and  schedule  variances  were  confuted  using 
the  following  formulas: 

Cost  Variance  (CV)  =  BCWP  -  ACWP  (4) 

Schedule  variance  (SV)  =  BCWP  -  BCWS  (5) 

Similarly,  mean  schedule  variances  were  calculated. 

3.  The  cost  and  schedule  variances  in  percentages  for 
each  entry/row  in  the  datcibase  were  computed  using  the 
following  formulas: 

%  Cost  Variance  (CV%)  =  (CV+bCWP)*100  (6) 

%  Schedule  Variance  (SV%)  =  (SV+BCWS)*100  (7) 

4.  For  each  contract  the  number  (n)  of  cost  variance  in 
percent  entries  in  each  stage  were  summed  (an  entry  is 
generated  each  time  a  contractor  reports  cost  or  schedule 
information  to  the  government).  This  summed  value  was 
divided  by  n  to  calculate  a  mean  cost  variance  in  percent  for 
each  stage  of  each  contract  using  the  formula: 

CV%)l(by  stage  by  contract) 

”(5i  CV%(by  stage  by  contract))  ^  (8) 
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Variances  cire  converted  into  percentages  to  normalize  the 
data  for  analysis.  By  converting  to  percentages,  contracts 
that  cure  monetcurily  large  do  not  have  excessive  influence  on 
the  analytical  results. 

5.  Each  contract  was  identified  by  service,  phase,  type 
and  stage.  This  resulted  in  48  distinct  groupings.  The 
resulting  969  lines  of  data  from  the  above  steps  are  listed 
in  Appendix  B. 

Benchmark  Calculations 

The  mean  of  each  of  the  48  cost  variance  and  48  schedule 
variance  groupings  was  computed  by  summing  the  number  (n)  of 
cost  or  schedule  variances  in  each  category  and  dividing  by 
n,  as  follows: 

p.CV%(by  category)  *  (E  CV%(by  category))  ^  (9) 

(iSV%(by  category)  “  (5!  SV%(by  category))  (10) 

The  benchmarks  are  in  the  Analysis  Section  (chapter  IV). 

Flowchart  of  the  Analysis 

The  flowcheurt  on  the  following  page  depicts  the  core 
decision  steps  used  in  the  analysis  ( Figure  1 ) . 

Analysis  of  Variance  (ABOVA) 

Overview.  The  ANOVA  approach  was  used  to  test  the 
hypotheses  to  determine  v^ether  there  is  a  statistically 
significant  difference  between  related  groupings.  The  ANOVA 


32 


Figure  1 
Flowchart 
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design  appropriate  for  this  study  is  the  fixed  factor  level 
ANOVA  model  (Neter  and  others,  1990:528). 

This  study  incorporates  a  four  factor  nested  design.  It 
is  an  unbalanced  design  (unequal  sample  sizes)  with  one 
category  containing  no  data  points  (the  Air  Force, 

Production,  Cost  ReimburscUble,  1st  Stage  category).  The 
nested  design  explicitly  considers  the  interrelationships  of 
the  multiple  factors  in  the  analysis. 

AHOVA  Terminology.  The  following  discussion  relates 
the  standard  ANOVA  terminology  to  the  attributes  of  this 
study  (Neter  and  others,  1990:522-525). 

Type  of  study.  This  study  is  based  on 
observational  data  from  the  DAES  database,  as  a  consequence, 
the  factors  under  cuialysis  may  be  called  classification 
factors. 

Factors.  There  are  four  factors  (independent 
vcuricUDles)  in  the  study.  They  are:  1)  the  branch  of  military 
service,  2)  the  phase  the  contract  applies  to,  3)  the  type  of 
contract  and  4)  the  stage  of  the  contract. 

Factor  Levels.  Each  of  the  four  factors  in  the 
study  are  divided  into  the  appropriate  number  of  collectively 
exhaustive  and  mutually  exclusive  factor  levels.  The  branch 
of  military  service  factor  is  divided  into  three  factor 
levels  consisting  of.  Air  Force,  Army  or  Navy.  The  phase  the 
contract  applies  to  is  divided  into  two  levels,  development 
or  production.  The  type  of  contract  is  divided  into  two 


34 


levels,  cost  reiinburse±>le  or  fixed  price.  The  stage  of  the 
contract  is  divided  into  four  levels,  1st  quarter,  2nd 
quarter,  3rd  quarter  or  4th  quarter.  Figure  2  is  an  exan^le 
of  a  four  factor  nested  design  with  multiple  factor  levels  as 
described  above. 

Multi  factor  Study.  This  is  a  multifactor  study 
containing  the  four  factors  (independent  varicd)les) 
previously  discussed.  A  diagram  of  one  of  the  nested  designs 
used  in  these  analyses  in  included  on  the  following  page 
(Figure  2). 

Type  of  Factors.  The  four  factors  in  this  study 
are  qualitative  and  nominal  in  nature. 

Dependent  Variables.  There  are  two  dependent 
varied>lea  in  this  study,  cost  variance  (in  percent)  and 
schedule  variance  ( in  percent ) . 

Treataents.  The  treatments  are  those  "buckets” 
appearing  at  the  bottom  of  the  nested  design.  Each  treatment 
is  a  unique  combination  of  the  factor  levels  cUsove  it.  The 
effect  each  treatment  has  on  the  dependent  varicd>le  will  be 
analyzed.  In  this  four  factor  study,  with  factor  levels 
consisting  of  three,  two,  two  and  four  categories  (reference 
factor  level  discussion  beginning  on  the  previous  page),  the 
total  number  of  treatments  in  the  bottom  factor  will  equal  48 
(3*2*2*4*48)  for  cost  variance  in  percent  and  another  48  for 
schedule  variance  in  percent. 
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Figure  2.  Four  Factor  Nested  Design  with  Managing  Service  as  the  Treatment 


AHOVA  Fixed  Factor  Level  Model.  The  reason  the 
AMOVA  fixed  factor  level  model  is  appropriate  for  this  study 
is  because  the  interest  is  in  the  specific  factor  levels 
contained  in  the  analysis.  This  is  in  contrast  to  a  random 
factor  level  model  where  inferences  are  desired  for  factor 
levels  not  included  in  the  model.  For  example,  if  the 
objective  of  this  study  was  to  make  inferences  not  of  just 
Air  Force,  Army  and  Navy  contracts  but  of,  say,  NASA, 
Department  of  Energy  and  Depaurtment  of  Commerce  contracts, 
then  the  random  factor  level  model  would  be  appropriate.  The 
ANOVA  fixed  factor  level  model  is  very  robust  against  certain 
departures  from  the  assumptions  of  the  model  such  as 
normality  and  constant  error  variance  (Neter  and  others, 
1990:607,  623). 

AHOVA  Rested  Design.  This  is  a  nested  design 
since  a  given  cost  or  schedule  variance  (in  percent)  is 
associated  with  a  specific  combination  of  factor  levels 
(service,  phase  of  contract,  type  of  contract  and  stage  of 
contract).  This  is  in  contrast  to  the  other  major  type  of 
design,  the  crossed  design,  where  a  given  treatment  could  be 
associated  with  any  combination  of  factor  levels  (Neter  and 
others,  1990:970-973). 

Testing  the  Prisuirj  Assumptions  of  AHOVA 

Determination  of  normality.  Initially  the  Shapiro- 
Wilk  statistical  test  was  used  to  determine  the  degree  of 
normality  of  the  data,  in  addition,  the  normal  probability 
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plot  was  created  and  visually  analyzed  bearing  in  mind  the 
IONOVA  fixed  factor  level  model  is  a  robust  model  and  is  valid 
for  moderate  departures  from  normality  (Neter  and  others, 
1990:623). 

Analysis  of  Nonconstancy  of  Error  Variance.  The 

Bartlett  statistical  test  is  appropriate  for  testing  the 
degree  of  constancy  of  the  error  variance  when  the  san^le 
sizes  are  unequal.  The  Bartlett  test  was  applied  in  two 
stages.  First,  the  entire  data  set  was  tested  for  equality 
of  error  vauriance.  Second,  for  each  group  of  means  for  idiich 
the  ANOVA  F-test  indicated  statistically  significant 
differences,  the  test  was  applied  to  the  mecuis  in  that  group 
to  further  explore  the  constancy  of  the  error  variance. 

First,  the  Bartlett  test  was  applied  to  the  entire  data 
set  to  examine  the  overall  constancy  of  the  error  variances. 
In  addition,  a  plot  of  the  residuals  was  analyzed.  "When  the 
error  variance  is  constant,  these  residual  plots  should  show 
about  the  Scune  extent  of  scatter  of  the  residuals  around  0 
for  each  factor  level"  (Neter  emd  others,  1990:610).  The 
plot  of  the  residuals  indicates  if  heteroscedasticity  is 
present  in  the  data.  As  with  departures  from  normeLLity,  the 
ANOVA  fixed  factor  level  model  is  a  robust  model  and  is  valid 
even  for  a  moderate  degree  of  unequal  error  veiriance. 

Second,  for  those  groups  containing  statistically 
significcuitly  different  meems,  the  Bartlett  test  was  applied 
to  that  specific  group  of  means.  This  more  detailed  analysis 
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of  the  error  variance  is  necessary  when  interpreting  the 
results  of  the  €Lnalysis.  If  the  error  variances  are  not 
equal,  then  the  probability  of  a  type  I  error  is  increased. 
Thus,  if  heteroscedasticity  is  present  and  if  a  result  is 
"borderline”  (indicating  the  null  hypothesis  should  be 
rejected — but  just  barely),  then  caution  must  be  exercised 
before  concluding  the  null  hypothesis  should  indeed  be 
rejected. 

Independence  of  the  Error  Terns.  Since  this  study 
is  not  based  on  tine  series  data,  the  error  terms  are  assumed 
to  be  independent  and  a  test  of  their  independence  is 
unnecesscoy. 

Hypotheses  Testing 

The  ANOVA  design  was  nested  each  of  three  different  ways 
during  the  study  to  get  the  appropriate  treatment  under 
analysis  at  the  bottom  of  the  design.  After  the  first 
nesting  produced  the  factor  military  service  at  the  bottom  of 
the  nest,  the  test  of  the  hypothesis  associated  with  testing 
among  military  services  was  conducted  (reference  the 
beginning  of  this  chapter  for  a  description  of  the  six 
hypotheses  in  this  study) .  The  nesting  in  the  design  was 
restructured  two  more  times  to  allow  testing  between  groups 
of  means  categorized  by  contract  type  xuid  then  by  progreun 
phase.  Thus,  this  analysis  tested  meeuis  three  different  ways 
through  three  variations  of  the  nested  design,  as  follows: 
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1.  Holding  program  phase,  contract  type,  and  contract 
stage  consteuit  the  branch  of  military  sexrvice  was  analyzed. 
Means  were  calculated  and  tested  to  determine  \diether  they 
had  statistically  significant  differences.  For  example,  the 
meeui  for  production,  fixed  price  contracts  from  the  first 
quarter  in  the  Air  Force  was  tested  against  the  mean  for 
production,  fixed  price  contracts  from  the  first  gucurter  in 
the  Army. 

If  the  null  hypothesis  is  rejected,  there  are 
statistically  significant  differences  between  the  cost  and/or 
schedule  variance  mecuis  in  percent  for  contracts  managed  by 
the  various  services. 

2.  Similarly,  holding  militeury  service,  program  phase, 
and  contract  stage  constant,  the  contract  type  was  varied  and 
means  were  calculated  and  tested. 

If  the  null  hypothesis  is  rejected,  there  are 
statistically  significant  differences  between  the  percentage 
cost  and/or  schedule  varieuice  means  for  the  two  contract 
types. 

3.  Lastly,  holding  military  service,  contract  type  and 
contract  stage  constant,  progreun  phase  was  veucied  euid  means 
were  calculated  £md  tested. 

If  the  null  hypothesis  is  rejected,  there  are 
statistically  significant  differences  between  the  percentage 
cost  and/or  schedule  veuriance  means  for  the  two  program 
phases . 
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Three  Steps  in  the  Analysis.  There  are  three 
specific  steps  in  the  ANOVA  analysis,  as  follows: 

1.  The  first  step  is  to  determine  if  there  are  any 
groups  that  contain  means  that  have  statistically  significant 
differences.  The  ANOVA  F-test  is  used  to  make  this 
determination . 

2.  The  second  step  is  to  determine  which  groups  of 
means  contain  the  means  that  are  statistically  significantly 
different.  An  ANOVA  F>test  is  used  via  the  SAS  CONTRAST 
statement  to  pinpoint  the  groups  of  means  requiring  further 
analysis . 

3.  The  third  step  is  to  determine  the  specific  means  in 
each  group  that  had  been  identified  as  containing 
statistically  significantly  different  means.  The  SAS 
ESTIMATE  Statement  is  used  as  part  of  this  analysis.  The 
Tukey-Kramer  multiple  coiq)arison  of  means  method  is  used  in 
this  final  step. 

Step  1  Determining  if  Any  Groups  Contain 
Statistically  Significantly  Different  Means.  This 
ANOVA  F-test  is  the  first  step  in  the  three  step  process  in 
determining  vdiere  statistically  significant  differences 
between  means  lie.  The  F-test  was  performed  to  indicate  if 
there  is  a  statistically  significant  difference  among  any  of 
the  groups  of  treatments  in  any  of  the  factor  levels.  The 
ANOVA  F-test  eliminates  the  need  to  test  each  and  every 
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treatment  in  the  factor  level  to  determine  if  statistically 
significant  differences  among  the  means  exists. 

Since  the  treatments  contained  in  the  factor  level  at 
the  bottom  of  the  nest  are  of  interest  in  this  study,  the  F- 
test  result  for  that  factor  level  was  analyzed  to  see  if  a 
more  detailed  analysis  was  necessary.  The  F-test  only 
indicates  if  there  is  a  statistical  difference  among  any  of 
the  groups  of  means  at  the  bottom  of  the  nest.  It  does  not 
indicate  which  of  the  means  £Lre  different.  A  more  specific 
CQiq>arison  of  means  would  need  to  be  conducted  to  determine 
which  groups  of  means  are  different. 

If  the  F-test  result  was  not  significant,  a  more 
detailed  analysis  was  not  necessary  and  steps  two  and  three 
were  not  accomplished.  The  alpha  value  used  to  determine 
statistical  differences  was  .05.  Thus,  if  the  F-test  for  the 
bottom  factor  level  in  the  nest  produced  a  p-value  greater 
than  .05,  there  was  not  a  statistically  significant 
difference  in  any  of  the  groups  of  means  in  that  factor  level 
and  no  further  analysis  was  accoo^lished. 

However,  if  the  F-test  for  the  bottom  factor  level  in 
the  nest  produced  an  p-value  equal  to  or  less  than  .05,  then 
further  analysis  was  needed  to  determine  which  of  the  groups 
of  means  contains  individual  oeans  vdiich  have  statistically 
significeuit  differences. 

Step  2  Analyzing  Which  Groups  Contain 
Statistically  Significantly  Different  Means.  Based  on 
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the  results  of  the  ANOVA  F-test  in  step  1  a  con?)arison  of 
means  may  be  warranted.  If  the  first  F-test  indicated  there 
was  a  statistically  significant  difference  somewhere  in  the 
groups  of  means  at  the  bottom  of  the  nest/  every  group  of 
means  was  analyzed  to  see  which  group(s)  of  means  contained 
‘  the  statistically  significant  difference.  This  is  the  second 
step  of  the  three  stage  process  to  locate  the  means  which 
have  statistically  significant  differences. 

Again,  the  F-test  was  used  to  pinpoint  the  group(s)  of 
means  requiring  further  analysis.  The  CONTRAST  statement  in 
SAS  was  used  to  analyze  the  group  of  means  in  each  factor 
level.  An  F-ratio,  and  associated  p-value,  was  produced  for 
each  group  of  means.  As  with  the  F-test  results  for  the 
factor  as  a  vdiole  (discussed  in  step  1),  a  p-value  equal  to 
or  less  than  .05  was  used  to  indicate  statistically 
significant  differences  among  some  of  the  means  in  that 
group. 

If  the  F-test  results  indicated  statistically 
significeuit  differences  among  some  of  the  means  in  a  group,  a 
con^arison  of  the  individual  means  within  each  group  was 
acconqplished.  Thus,  the  third  and  final  step  was  to 
determine  which  individual  means  within  a  group  have 
statistically  significant  differences. 

Step  3  Comparison  of  Specific  Means.  If  the 
F-test  in  step  2  indicated  the  factor  level  contained 
statistically  significant  differences  between  two  or  more 
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means,  a  comparison  betvieen  each  applicable  mean/treatment 
was  made.  The  ESTIMATE  statement  in  SAS  was  used  to  pinpoint 
the  specific  means  that  have  statistically  significemt 
differences.  This  ESTIMATE  statement  produced  a  T-value  for 
each  pair  of  means. 

The  Tukey-Kramer  multiple  con^arison  method  was  used  to 
determine  which  means  cire  statistically  significeuitly 
different.  This  variation  of  the  basic  Tukey  method  was  used 
because  of  the  unequal  saii^le  sizes.  The  Tukey-Kramer  method 
provides  protection  against  the  inferences  in  cases  where 
more  than  two  treatment  means  are  being  compared. 
Specifically,  when  conp2u:ing  the  mecuis  associated  with  the 
three  military  services  a  three-way  multiple  comparison  is 
necessary. 

The  T-values  produced  using  the  SAS  ESTIMATE  command 
were  compared  to  a  critical  value  to  determine  if  a 
significant  statistical  difference  exists  between  the  means. 
This  critical  value  was  extracted  from  a  Percentiles  of  the 
Studentized  Range  Distribution  table  (Meter  and  others, 
1990:1149).  Since  this  study  uses  a -.05,  the  tcQ^le  containing 

critical  values  associated  with  a  .95  family  confidence  level 
was  used. 

There  are  two  entering  arguments  for  the  Percentiles  of 
the  Studentized  Range  Distribution  table:  1)  the  number  of 
means  to  compare  (r)  and  2)  the  number  of  error  degrees  of 
freedom  ( v) .  The  value  of  r  will  be  2  or  3  depending  on  the 
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number  of  means  being  compared  (r  =  2  when  means  associated 
with  program  phase  or  contract  type  are  being  studied  and  r  = 
3  when  the  means  associated  with  military  service  is  being 
studied).  The  value  of  v  is  922  as  reflected  in  the  ANOVA 
tcdsle.  Given  this  rather  large  v  value,  the  critical  value 
extracted  from  the  tcd>le  is  3.31. 

Any  T-value  resulting  from  a  con^arison  of  means  that  is 
greater  thcui  3.31  indicates  a  significant  statistical 
difference  exists  between  those  means. 

The  results  of  the  above  procedure  are  contained  in 
Chapter  IV. 
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IV.  Analysis 


bion  of  Cost  and  Schedule  Variance  Means 
!  data  from  the  DAES  database  were  divided  into  48 
.es.  Cost  and  schedule  variance  means  were  calculated 
ibed  in  Chapter  III.  This  created  the  96  benchmarks 
id  in  T2d>le  2  on  the  next  page.  Information  in  the 
:  broken  down  by  military  service,  phase  of  contract, 
contract  euid  stage  of  contract.  The  number  of 
:s  in  each  category /treatment,  the  mean  cost  and 
}  variances  in  percent,  and  the  standard  deviation 
:ed  with  each  mean  is  presented.  This  quick  reference 
in  be  used  by  program  managers  to  compare  historical 
performance  with  their  specific  program’s 
ince. 

the  AHOVA  Assumption  of  Normality 

3  results  of  the  Shapiro-Wilk  statistical  test  were 
Dr  the  cost  variance  in  percent  and  .8507  for  the 
3  variance  in  percent.  Although  these  results 
3  the  distributions  are  not  exactly  normal,  the 
are  strong  enough  to  conclude  the  ANOVA  assumption 
oality  is  being  met. 

e  assumption  of  normality  was  discussed  with  Professor 
E.  Reynolds,  a  resident  statistician  at  the  Air  Force 
te  of  Technology.  Professor  Reynolds  stated  when 
with  a  large  Scunple  (969  in  this  case)  the  Shapiro- 
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Wilk  test  will  rarely  uncategorically  indicate  the  data  are 
strongly  normally  distributed.  The  adverse  inqpact  that  a 
large  sample  size  had  on  the  results  of  the  normality  test 
dictated  judgment  be  exercised  in  determining  whether  the 
assunption  of  normality  was  satisfied.  The  ANOVA  model, 
pcurticulcirly  the  fixed  factor  level  model,  is  robust  against 
departures  from  normality  (Neter  euid  others,  1990:  607). 

Normal  probability  plots  of  the  residuals  for  cost 
varicuice  in  percent  and  for  schedule  variance  in  percent  were 
created  to  provide  a  visual  indication  of  the  distribution  of 
the  data.  Analyses  of  these  plots  indicated  the  data  did  not 
seriously  depcirt  from  normality,  and  therefore  the  normality 
assuoption  was  satisfied  (Neter  and  others,  1990:613-614). 

The  normal  probability  plots  for  cost  and  schedule  variance 
in  percent  are  contained  in  Appendices  C  and  D,  respectively. 

Testing  the  AHOVA  Assumption  of  Equality  of  the  Error 
Variance  for  the  Entire  Data  Set 

When  applied  to  the  entire  data  set,  the  results  of  the 
Beurtlett  statistical  test  for  equality  of  the  error  variances 
for  cost  v€u:iance  in  percent  and  schedule  vcuriance  in  percent 
were  4.55  and  18.98,  respectively.  Unfortunately,  the 
Bartlett  test  is  sensitive  to  even  small  departures  from 
normality.  Thus,  the  more  the  data  depart  fr<xn  perfect 
normality  the  less  conclusive  are  the  results  of  the  Baurtlett 
test.  In  this  case,  a  better  way  of  analyzing  the  equality 
of  error  variances  is  by  using  a  plot  of  the  residuals. 
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The  assuiq>tion  of  constant  error  variance  for  all  factor 
levels  was  tested  by  plotting  the  residuals  against  their 
predicted  values  for  cost  variance  in  percent  and  for 
schedule  variance  in  percent.  The  analysis  of  these  plots 
indicated  that  the  error  terms  have  nearly  constant  variance, 
therefore,  this  l^OVA  assuoqption  is  also  satisfied  (Neter  and 
others,  1990:609-611).  The  constant  error  variance  plots  of 
the  residuals  for  cost  and  schedule  veiriance  in  percent  are 
in  Appendices  C  and  O,  respectively. 

Overall  Assessment  of  the  ANOVA  Assumptions 

Based  on  the  statistical  test  results  and  a  visual 
analysis  of  the  plots,  the  normality  assumption  and  the 
constant  error  varicuice  assumption  have  been  adequately  met. 

Running  the  Heated  AMOVA  Model  (Step  1) 

The  ANOVA  model  was  run  six  different  times  to  test  the 
six  null  hypotheses  (three  times  for  cost  variance  in  percent 
and  three  times  for  schedule  vcuriance  in  percent).  The  ANOVA 
statistical  results  with  cost  variance  in  percent  as  the 
dependent  variable  and  schedule  variance  in  percent  as  the 
dependent  veuriable  ^u:e  summcirized  in  Tables  3  and  4, 
respectively  (note:  the  overall  statistical  results  for  the 
entire  model  are  presented  followed  by  the  statistical 
results  for  each  of  the  three  ways  the  model  was  nested— of 
critical  iiq>ortance  is  the  p-value  associated  with  the  last 
factor  in  each  of  the  nested  models). 
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Table  3 


ANOVA  Model  Results  with 

Cost  Vcuriance  in  Percent  as  the  Dependent  variable 


Source  of  Degrees  of 
Variation  Freedom 


Model 


Error 


Total 


Phase 


F-Value  1  P-Value 


1.83 


2.03 


4.90 


1.13 


2.51 


0.80 


2.86 


1.12 


1.03 


1.35 


1.85 


1.26 


0.1766 


0.1317 


0.0001 


.2866 


0.1133 


0.5229 


0.0001 


.3106 


0.3095 


0.2485 


0.0168 


.1881 


Tcible  4 

ANOVA  Model  Results  with 

Schedule  Variance  in  Percent  as  the  Dependent  Variable 


Mean 

Squares 


F-value  P-Value 


Summary  F-statistics  are  shown  in  Table  5  and  describe 
the  overall  assessment  of  statistical  difference  in  the  means 
contained  in  the  most  deeply  nested  factor  in  each  one  of  the 
six  ANOVA  models.  These  F-statistics  pertain  to  the  first 
step  in  the  three  sequence  step  to  determine  which  means, 
among  groups  of  means,  are  statistically  different. 


Tcible  5 

F-Test  Summary  Table  for  Alpha  =  .05  (Step  1) 


Phase 

Type 

cv  % 

D-value 

0.2866 

0.3106 

0.1881 

Cannot  Reiect 

Cannot  Reiect 

sv  % 

D-value 

0.0001 

0.9546 

0.1122 

Ho:  All  )is  Equal 

Reiect 

Cannot  Reiect 

Cannot  Reiect 

Statistically  significant  differences  among  the  means 
only  exists  in  the  schedule  variance  in  percent  model  with 
militcUY  service  as  the  bottom  factor  in  the  nest  (the  p- 
value  associated  with  the  F-ratio  was  0.0001  which  is  smaller 
them  the  .05  alpha  level  used  in  the  study).  The  other  p- 
values  were  greater  than  .05  which  indicated  a  statistically 
significant  difference  does  not  exist  within  the  groups  of 
means.  Thus,  Step  2  of  the  analysis  was  only  performed  on 
each  of  the  16  factor  levels  (groups  of  means)  at  the  bottom 
of  the  nested  design  where  military  service  is  the  factor. 
Figure  2  illustrates  this  nested  design. 
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F-t«8t  on  Military  Service  Factor  Levels  for  Schedule 
variance  (Step  2) 

This  is  the  second  step  in  the  three  step  sequence  to 
determine  \diere  the  statistically  significant  differences  in 
the  means  lie.  This  step  is  necessitated  by  the  first  F-test 
indicating  there  is  a  statistical  difference  somewhere  in  the 
military  services  factor  for  schedule  variance.  The  CONTRAST 
statement  in  SAS  was  used  to  produce  the  F~ratios,  and  the 
associated  p-vcJ.ues,  for  each  factor  level.  Thus,  each  of 
the  16  factor  levels  were  euialyzed  to  pinpoint  which  group(s) 
of  means  are  statistically  significcuitly  different.  The 
results  for  the  16  factor  levels  are  in  Table  6  below. 


Table  6 


Results  of  F-Test  Analyzing  Groups  of  Meeuis  (Step  2) 


Phase  of 
Contract 

Type  of  Contract 

Stage  of 
Contract 

P-Value 

Development 

Cost  Reimbursable 

1st  Quarter 

0.6473 

2nd  Qucurter 

0.5672 

0.4744 

0.7051 

Fixed  Price 

1st  Quarter 

0.6088 

2nd  Quarter 

0.7273 

0.7201 

0.9969 

Production 

Cost  Reimbursable 

1st  Quarter 

2nd  Queurter 

0.5605 

3rd  Quarter 

0.9147 

4th  Qu£ucter 

0.9408 

Fixed  Price 

1st  Quarter 

0.0001 

2nd  Qu£urter 

0.0001 

3rd  Quarter 

0.0001 

4th  Quarter 

■[•R4IWI 
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Additional  Testing  of  the  AHOVA  Assumption  of  Equality 
of  the  Error  Variances 

For  the  groups  of  means  with  statistically  significantly 
different  means,  the  Bartlett  test  was  applied  to  each  group 
to  evaluate  the  extent  of  euiy  heteroscedasticity  that  might 
be  present.  If  a  significant  amount  of  heteroscedasticity  is 
present  the  results  must  be  interpreted  with  caution.  The 
results  of  the  Bartlett  test  for  specific  group  means  are 
contained  in  Table  7  below.  The  test  results  along  with  the 
plots  of  the  raw  data  illustrating  the  dispersion  about  each 
mean  are  in  Appendix  E. 


Table  7 

Bartlett  Test  Results  for  Equality  of  Error  Variance  for 
Groups  of  Means  with  Statistically  Significcint  Differences 


Program 

Phase 

Type  of 
Contract 

Stage  of 
Contract  . 

Bartlett 

Test 

Result 

Equal 

Error 

Variances? 

Production 

Fixed  Price 

1st  Quarter 

0.00013 

NO 

2nd  Quarter 

0.39518 

YES 

0.18788 

YES 

Since  the  Bartlett  test  did  not  indicate  the  error 
Vciriances  for  production,  fixed  price,  1st  quarter  contracts 
were  equal,  ^ul  analysis  of  the  associated  scatter  plot  is 
warranted.  Analysis  of  the  scatter  plot  indicated,  with  the 
exception  of  one  outlier  Navy  contract,  the  data  are 
homoscedastic .  Thus,  the  assumption  of  the  equality  of  the 
error  variance  is  satisfied. 
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Coaparlson  of  Means  Hlthln  Groups  Identified  as 
Containing  Statistically  Significantly  Different  Means 
(Step  3) 

Based  on  the  results  of  the  F-test  of  military  service 
factor  levels  for  schedule  variance  (Tcd^le  4),  there  are 
three  groups  that  contain  statistically  significantly 
different  means  (p-value  equal  to  or  less  than  .05).  Thus, 
an  actual  coiif>arison  of  the  means  within  each  of  those  groups 
Weus  warranted  using  the  Tukey-Kramer  multiple  comp2u:ison 
test.  This  is  the  third  step  in  the  three  step  analysis 
previously  discussed. 

The  ESTIMATE  command  in  SAS  produced  T-values  for  each 
of  the  possible  combinations  of  means  within  each  treatment 
group  identified  as  containing  statistically  significantly 
different  means.  Each  T-value  was  con^cured  to  the  critical 
value  of  3.31.  This  value  was  extracted  from  the  appropriate 
Percentiles  of  the  Studentized  Range  Distribution  table  as 
previously  discussed  in  the  Methodology  Chapter. 

The  specific  means  that  are  statistically  significantly 
different  are  identified  in  Table  8  on  the  following  page. 
Although  the  critical  value  used  in  the  study  was  3.31  (a  - 
.05),  two  other  critical  values  associated  with  other  alpha 
levels  €u:e  presented  for  reference  only  when  interpreting  the 
overall  significance  of  the  results  in  Table  8,  as  follows: 

1)  the  critical  value  at  a  .01  is  4.12  and,  2)  the  critical 
value  at  a  »  .10  is  2.90. 


54 


Table  8. 


Results  of  the  Tukey-Kramer  Test  Between  Statistically 
Significantly  Different  Treatment  Means  (Step  3) 


Category 

Treatments 

Absolute 

T-value 

Reject  Ho  given  a 
3.31  critical  value? 

Production 

Fixed  Price 

1st  Quairter 

Air  Force 
versus 
Army 

1.18 

NO 

Air  Force 
Versus 
Navy 

6.89 

YES 

Production 
Fixed  Price 

1st  Qucurter 

Am^ 

Versus 

Navy 

7.29 

YES 

Production 
Fixed  Price 

2nd  Quarter 

Air  Force 
Versus 
Army 

5.11 

YES 

Air  Force 
Versus 
Navy 

2.53 

NO 

Production 

Fixed  Price 

2nd  OueLTter 

Army 

Versus 

Navy 

7.06 

YES 

Production  - 
Fixed  Price 

3rd  Quarter 

Air  Force 
Versus 
Army 

4.05 

YES 

Production 
Fixed  Price 

3rd  Quarter 

Air  Force 
Versus 
Navy 

1.31 

NO 

Production 
Fixed  Price 

3rd  Quarter 

Army 

Versus 

Navy 

4.98 

YES 

Analytical  Sumaary 

Of  the  96  cost  and  schedule  variance  means  (in  percent) 
benchmarks  that  were  analyzed,  six  tested  statistically 
significcuitly  different.  The  six  means  (in  percent)  that 
tested  different  were  confined  to  the  production,  fixed  price 
category. 
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V 


Conclusions 


Discussion 

This  thesis  used  data  in  the  DAES  database  to  calculate 
96  benchmarks.  These  benchmarks  can  be  used  by  Department  of 
Defense  Program  Managers  for  coo^arison  when  evaluating  the 
cost  cuid  schedule  status  of  their  program  contracts. 

The  96  benchmarks  are  comprised  of  48  cost  variance 
percentage  meeuis  and  48  schedule  V2u:iance  percentage  means. 
These  48  categories  of  cost /schedule  variance  percentage 
means  are  a  result  of  classifying  each  data  point  in  the 
refined  DAES  datcdsase  into  the  following  collectively 
exhaustive,  mutually  exclusive  categories: 

1.  As  supporting  either  the  development  phase  of  a 
program  or  the  production  phase  of  a  program 

2.  As  either  a  cost  reimbursable  type  contract  or  a 
fixed  price  type  contract 

3.  As  representing  contract  data  from  the  1st  quarter, 
2nd  quarter,  3rd  quarter,  or  the  4th  quarter  of  the  contract 

4.  As  managed  by  the  Air  Force,  Army,  or  Navy. 

The  two  program  phases  times  the  two  contract  types  times  the 
four  contract  stages /quarters  times  the  three  managing 
services  equals  48  cost  and  48  schedule  benchioarks. 

These  benchmeurks  were  nested  using  the  ANOVA  fixed 
factor  model.  The  cost  and  schedule  means  in  each  of  the 
three  various  nested  designs  were  tested  for  statistically 
significant  differences  between  treatments. 
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Liaitation 

The  sanqple  data  did  not  test  to  be  exactly  normally 
distributed  nor  test  to  have  exactly  constant  error  variance. 
Thus,  judgment  was  exercised  to  conclude  the  population  was 
normally  distributed  with  constant  error  variance  to  the 
degree  required  by  the  ANOVA  fixed-factor  level  model. 

Corclu  iions 

Not  one  of  the  statistical  tests  of  cost  variance  means 
(in  percent)  yielded  statistically  significant  differences. 
Only  six  of  the  48  schedule  variance  means  (in  percent) 
yielded  statistically  significant  differences.  The  small 
number  of  statistically  significeuit  differences  between  the 
iBftana  was  partly  due  to  the  somewhat  small  sample  sizes  in 
the  study  and  the  rather  large  variances  which  yielded  large 
confidence  intervals. 

Large  standard  deviations  in^acted  the  S-curves  that 
Knepp  and  Stroble  produced  in  their  thesis  (reference  the 
literature  review  in  chapter  two).  Knepp  and  Stroble  found 
that,  given  the  large  variances /standard  deviations  in  the 
data,  their  tailored  S-curves  had  limited  applicability. 

Similarly,  the  large  standard  deviations  surrounding  the 
data  dictate  using  caution  when  applying  the  tailored  cost 
euid  schedule  benchmarks.  Program  managers  must  consider  the 
associated  standard  deviation  when  using  a  benchmark  in 
gauging  the  status  of  their  programs. 


Conclusions  relating  to  the  individual  factors  are: 

1.  When  tested  across  program  phases,  it  was  determined 
there  are  no  statistically  significant  differences  between 
the  cost  varicuice  percentages  of  development  contracts  and 
production  contracts,  similarly^  there  are  no  statistically 
significeuit  differences  between  the  schedule  varieuice 
percentages  of  development  contracts  and  production 
contracts . 

2.  When  tested  across  contract  type,  it  was  also 
determined  there  are  no  statistically  significant  differences 
between  the  cost  variance  percentages  of  cost  reimbursable 
contracts  and  fixed  price  contracts.  Likewise,  there  are  no 
statistically  significeint  differences  between  the  schedule 
variance  percentages  of  cost  reimbursable  contracts  and  fixed 
price  contracts. 

3.  However,  vdien  tested  across  military  service,  there 
were  a  total  of  six  treatments  that  were  statistically 
significantly  different  in  their  schedule  variances  in 
percent..  Basically,  the  statistically  significant 
differences  between  the  military  services  are  confined  to  the 
production  phase,  fixed  price  type  contracts  in  the  first, 
second  and  third  stages /quarters. 

Bow  Cost  and  Schedule  Benchmarks  Can  be  Used  by 
Program  Managers 

Program  mcuiagers  should  use  the  cost  and  schedule 
benchmarks  that  this  study  produced  for  reference  only. 
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Statistical  testing  of  the  benchmarks  indicated  the  vast 
majority  of  them  \«ere  not  statistically  significantly 
different  from  one  another  (primarily  due  to  the 
con^aratively  Icirge  standcurd  deviation  associated  with  the 
benchmarks ) .  Thus ,  program  managers  should  not  use  these 
cost  and  schedule  benchmarks  to  conclude  the  status  of  their 
current  program  is  categorically  edx)ve  or  below  the 
historical  average  of  similair  contracts.  However,  by 
considering  the  historical  cost  and  schedule  benchmarks  for 
similcir  contracts  (caveated  by  the  standard  deviations) 
program  managers  can  get  a  ballpark  idea  of  how  their 
programs  are  performing. 

ReccMUiendations  for  Further  Research 

The  following  suggestions  for  future  research  are  made: 

1.  Since  some  categories  of  benchmarks  have 
statistically  significant  differences  in  the  means,  it  is 
recommended  that  further  study  be  done  to  try  to  determine 
the  reasons  for  these  differences.  Determining  why  one 
military  service  has  historically  had  different  cost  and 
schedule  variances  than  other  military  services  might  benefit 
DoD.  An  example  of  a  benefit  might  be  the  sharing  of 
managerial  techniques  found  conducive  to  minimizing  cost  and 
schedule  varicuices. 

2.  Since  some  categories  of  data  have  very  few  data 
points  (and  one  has  none — Air  Force,  production,  cost 
reimbursable,  1st  quarter  category) ,  it  is  recommended  that  a 


59 


follow-up  study  be  done  in  a  few  years  after  the  DAES 
database  has  accumulated  more  data  to  yield  larger  sample 
sizes.  Additional  data  points  in  each  category  might  narrow 
the  confidence  intervals  eiround  the  individual  benchmarks, 
and  could  affect  the  results  of  the  hypothesis  testing  for 
statistically  significeuit  differences. 
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Appendix  A:  Definition  of  Terms 


1.  Actual  Cost  of  Work  Performed  (ACWP) .  "The  cost 
incurred  and  recorded  in  accomplishing  the  work  performed 
within  a  given  time  period"  (DoD  Instruction  5000.2/ 
1991:11-8-2-1) . 

2.  Benchmark.  "A  point  of  reference  from  which 
measurements  of  any  sort  may  be  made"  (Webster's,  1971:203). 

3.  Budgeted  Cost  of  Work  Performed  (BCWP) .  "The  sum 
of  the  budgets  for  completed  work  packages  and  completed 
portions  of  open  work  packages,  plus  the  applicable  portion 
of  the  budgets  for  level  of  effort  and  apportioned  effort" 
(DoD  Instruction  5000.2,  1991:11-8-2-1). 

4.  Budgeted  Cost  of  Work  Scheduled  (BCWS) .  "The  sum 
of  budgets  for  all  work  packages,  planning  packages,  etc., 
scheduled  to  be  accomplished  (including  in-process  work 
packages)  plus  the  amount  of  level-of-effort  and  apportioned 
effort  scheduled  to  be  accomplished  within  a  given  time 
period"  (DoD  Instruction  5000.2,  1991:11-8-2-1). 

5.  Cost  Overrun.  "The  amount  by  which  a  contractor 

exceeds  (a)  the  estimated  cost  and/or  (b)  the  final 
limitation  (ceiling)  of  his  contract"  (DoD  DSMC,  1991:8- 
23).  .  * 

6.  Contract  Budget  Base  (CBB) .  "The  negotiated 
contract  cost  plus  the  estimated  cost  of  authorized  unpriced 
work"  (DoD  Instruction  5000.2,  1991:11-8-2-2). 


7.  Cost  Perfomance  Report  (CPR) .  "A  monthly  DoD 
report  generated  by  the  contractor  to  obtain  cost  and 
schedule  status  information  for  program  management.  The  CPR 
is  intended  to  provide  early  identification  of  problems 
having  significant  cost  impact,  effects  of  management 
actions  and  program  status  information  for  use  in  making  and 
validating  management  decisions”  (Fleming,  1988:507). 

8.  Cost/Schedule  Control  Systems  Criteria  (C/SCSC) . 
"Government  established  standards  which  a  contractor's 
internal  management  system  must  meet  in  order  to  insure  the 
government  of  effective  planning  and  control  of  contract 
work"  (Fleming,  1988:507). 

9.  Cost  Variamce.  "Budgeted  Cost  of  Work  Performed 
minus  Actual  Cost  of  Work  Performed.  A  negative  cost 
variance  is  an  unfavorable  condition  and  indicates  a  cost 
overrun.  A  positive  cost  variance  indicates  a  cost 
underrun"  (Christensen,  1992:20). 

10.  Defense  Acquisition  Executive  Summary  (DAES) . 
"Defense  Acquisition  Executive's  principle  mechanism  for 
tracking  programs  between  milestone  reviews.  Includes 
programs  subject  to  the  Selected  Acquisition  Report  (SAR), 
and  any  non-SAR  programs  subject  to  review  by  the  Defense 
Acquisition  Board"  (DoD  DSMC,  1991:B-27). 

11.  Judgment  Sampling.  This  is  a  non-random  sampling 
method  which  relies  on  the  judgment  and  intuition  of  the 
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analyst  to  ensure  the  sample  accurately  reflects  the 
population. 

12.  Program  Manager  (PM).  “A  military  or  civilian 
official  who  is  responsible  for  managing  an  acquisition 
program"  (DoD  Instruction  5000.2  1991:15-14).  PMs  are  the 
Air  Force,  Army,  and  Navy  organizational  managers 
responsible  for  guiding  the  research,  development, 
production,  testing  and  fielding  of  a  new  weapon  system. 

13.  Schedule  Slippages.  These  occur  when  milestones 
are  not  met  on  time  and  the  program  is  delayed. 

14.  Total  Allocated  Budget  (TAB).  “The  sum  of  all 
budgets  allocated  to  the  contract.  Total  allocated  budget 
consists  of  the  performance  measurement  baseline  and  all 
management  reserve.  The  total  allocated  budget  will 
reconcile  directly  to  the  contract  budget  base.  Any 
differences  will  be  docmumented  as  to  quantity  and  cause" 
(DoD  Instruction  5000.2,  1991:11-3-2-3). 

15.  Unfavorable  Cost  or  Schedule  Variances.  A  variance 
is  unfavorable  when  incurred  costs  exceed  planned  costs  at  a 
given  point  or  the  contract  is  behind  the  planned  schedule. 

16.  Weapon  or  Materiel  System.  These  systems  are 
generally  defined  as  any  research,  development,  or  test  and 
evaluation  contract  over  $60  million  and  any  procurement 
contract  over  $250  million. 
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Appendix  B:  Data  Used  in  the  ANOVA  Nested  Models 


Air  Force 
Air  Force 


Air  Force 
Air  Force 
Air  Force 
Air  Force 
Air  Force 


Air  Force 


Air  Force 
Air  Force 


Havy 

Navy 

Army 

Army 

Army 

Army 


Contract 
Type 


rice 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Coat  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimburscdale 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 


Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Development 

Development 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Development 

Development 

Production 

Production 

Production 

Development 

Production 

Production 

Production 

Development 

Production 

Development 

Production 

Production 

Production 

Development 

Development 

Development 

Development 

Production 

Development 

Production 

Development 

Development 

Production 

Development 

Production 

Production 

Production 


Avg 

Avg 

C.V.% 

S.V.% 

2.13 

-4.08 

-2.44 

-2.09 

-2.56 

-3.83 

-21.09 

1.69 

10.71 

6.94 

-1.68 

-18.51 

-5.47 

-10.30 

-5.64 

9.33 

-25.57 

4.96 

-84.25 

-8.20 

3.90 

-17.20 

-5.67 

-7.22 

1.37 

-10.48 

20.33 

-17.32 

16.33 

-26.61 

0.00 

-19.17 

31.25 

-5.00 

-1.67 

-1.41 

-4.76 

-12.12 

7.14 

-15.71 

0.00 

0.00 

-21.19 

-13.08 

1.47 

-5.16 

-6.88 

-7.65 

0.93 

1.30 

-4.12 

-3.21 

6.45 

5.79 

0.92 

1.59 

37.70 

-3.71 

-0.47 

-5.67 

3.58 

-8.92 

-0.76 

-0.64 

7.83 

0.10 

-3.75 

-5.65 

-25.00 

-6.67 

33.33 

0.00 

5.21 

-15.04 

0.79 

0.00 

6.27 

-3.33 

2.38 

2.50 

10.23 

12.50 

-4.93 

-9.31 

-17.32 

-14.52 

-7.14 

-7.78 

-22.48 

29.76 

0.00 

-11.94 

-8.02 

0.00 

-12.28 

0.00 

0.00 
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Force 

Force 

Force 

Force 

Force 

Force 

Force 

Force 

Force 


Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Coat  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Coat  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbur'^able 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Coat  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 


veiopmenc 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Development 
Development 
Production 
Development 
Production 
Production 
Production 
Production 
Production 
Production 
Development 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Development 
Production 
Production 
Production 
Production 
Development 
Production 
Devel^ment 
Production 
Production 
Production 
Production 
Development 
Production 
Production 
Development 
Production 
Production 
Production 
Development 
Production 
Production 
Production 
Production 


-8.33 

14.29 

-16.67 

3.27 


-11 

.01 

0 

.00 

0 

.00 

-11 

.67 

0 

0 

.00 

3 

.89 

4 

.17 

-19 

.43 

-8 

.37 

-16 

.96 

0 

.00 

-33 

.33 

0 

.00 

0 

.00 

11 

.60 

3 

.20 

-0 

.54 

6 

.20 

-1 

.26 

-6 

.97 

6 

.91 

8 

.44 

2 

.35 

1 

.39 

3 

.33 

7 

.49 

4 

.35 

8 

.75 

1 

.39 

-1 

.51 

7 

.79 

-5 

.73 

-3 

.38 

-3 

.83 

0 

10.70 
-2.04 

6.20 

0.00 

0.00 

12.50 

4.08 

-9.35 

12.35 

0.00 

-1.00 

28.70 


21 

.43 

-3 

,13 

-22 

.22 

0 

.00 

-16 

.67 

-6 

.97 

-2 

.63 

-11 

.23 

-22 

.43 

-1 

.68 

-10 

,71 

-39 

.92 

-14 

.81 

0 

.00 

22 

45 

23 

97 

52 

59 

-6 

.09 

8 

.05 

-23 

.91 

4 

.93 

-1 

.85 

0 

.00 

-0 

.58 

0 

.00 

-16 

.67 

-4 

.46 

-1 

90 

-13 

.07 

-24 

.17 

19 

.36 

-10 

,65 

-7 

.31 

-13 

.47 

-7 

.28 

4 

.09 

-1. 

34 

-3. 

92 

-6. 

09 

-29 

.17 

-5 

.88 

-46 

.43 

19 

51 

-13 

.25 

-5 

74 

-23 

79 

-13. 

73 

5 

.81 
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urs 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 


veioproent 
Development 
Production 
Production 
Development 
Production 
Developmrat 
Development 
Development 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Production 
Development 
Development 
Production 
Production 
Development 
Development 
Development 
Development 
Development 
Development 
Development 
Development 
Development 
Development 
Production 
Production 
Production 
Development 
Development 
Development 
Devel<^ment 
Development 
Development 
Development 
Development 


0.72 

-12.98 

-7.51 

-24.88 

6.18 

-50.78 

14.65 

-13.97 

1.37 

0.00 

-0.69 

-14.90 

11.98 

-17.26 

-2.38 

-10.64 

-4.53 

133.52 

-6.08 

27.84 

1.19 

17.81 

50.83 

8.33 

5.26 

-32.14 

16.67 

-18.18 

16.67 

-20.00 

10.71 

-1.39 

4.57 

-14.58 

-1.11 

-3.71 

7.63 

0.00 

5.00 

25.00 

10.00 

-9.09 

0.00 

-7.14 

-3.45 

16.00 

13.03 

-13.75 

17.78 

-43.33 

0.00 

-9.09 

13.54 

-30.75 

4.76 

11.05 

-47.54 

1.39 

-19.84 

-2.27 

-5.71 

-7.84 

-8.93 

11.67 

-17.71 

5.87 

-38.57 

-3.57 

-13.39 

-1.45 

-8.00 

6.06 

-10.81 

15.74 

-17.53 

3.36 

-9.59 

-3.33 

-6.25 

0.00 

-12.82 

15.97 

-28.30 

-8.33 

-8.75 

2.27 

-7.50 

5.91 

-3.19 

-40.00 

0.00 

-2.29 

16.25 

0.00 

-1.52 

-13.78 

0.00 

0.00 

-20.00 

0.00 

-1.85 

-4.56 

-5.08 

-4.50 

0.00 

0.00 

4.03 

0.00 

Air  Force 

Fixed  Price 

Production 

- 1 - 

Air  Force 

Fixed  Price 

Development 

1 

5.26 

-8.93 

Air  Force 

Cost  Reimbursable 

Development 

1 

0.00 

-12.50 

Amy 

Fixed  Price 

Development 

1 

-14.84 

Amy 

Fixed  Price 

Production 

1 

2.28 

-1.47 

Amy 

Fixed  Price 

Production 

1 

7.08 

-2.98 

Amy 

Fixed  Price 

Production 

1 

12.50 

Air  Force 

Fixed  Price 

Production 

1 

0.00 

0.00 

Havy 

Fixed  Price 

Production 

1 

-33.33 

66.67 

Air  Force 

Cost  Reimbursable 

Developmient 

1 

1.39 

1.10 

Amy 

Cost  Reimbursable 

Development 

1 

0.00 

-6.67 

Air  Force 

Fixed  Price 

Production 

2 

3.85 

Air  Force 

Fixed  Price 

Production 

2 

0.00 

-2.62 

Air  Force 

Fixed  Price 

Production 

2 

6.30 

-6.41 

Air  Force 

Fixed  Price 

Production 

2 

0.00 

-4.35 

Air  Force 

Fixed  Price 

Production 

2 

-1.68 

2.37 

Air  Force 

Fixed  Price 

Production 

2 

0.00 

-5.78 

Amy 

Cost  Reimbursable 

Development 

2 

10.23 

-7.41 

Amy 

Fixed  Price 

Production 

2 

-1.67 

-2.19 

Amy 

Cost  Reimbursable 

Production 

2 

11.92 

-14.02 

Amy 

Fixed  Price 

Production 

2 

-2.39 

-19.16 

Army 

Fixed  Price 

Production 

2 

-0.35 

-16.99 

Amy 

Fixed  Price 

Production 

2 

0.08 

-8.01 

Army 

Cost  Reimbursable 

Production 

2 

3.75 

-4.76 

Army 

Cost  Reimbursable 

Production 

2 

-4.90 

-4.67 

Army 

Coat  Reimbursable 

Development 

2 

-0.45 

-5.82 

Army 

Cost  Reimbursable 

Developmient 

2 

-4.69 

-5.57 

Army 

Cost  Reimbursable 

Production 

2 

5.02 

-9.17 

Amy 

Cost  Reimbursable 

Development 

2 

9.09 

0.00 

Amy 

Cost  Reimbursable 

Developmient 

2 

-32.59 

-11.43 

Army 

Fixed  Price 

Production 

2 

19.19 

-19.15 

Army 

Fixed  Price 

Production 

2 

-4.32 

-51.92 

Azmy 

Fixed  Price 

production 

2 

7.14 

-17.65 

Amy 

Fixed  Price 

Production 

2 

7.14 

-14.58 

Navy 

Cost  Reimbursable 

Development 

2 

-6.96 

-3.58 

Navy 

Cost  Reimbursable 

Development 

2 

3.33 

-12.33 

Navy 

Fixed  Price 

Production 

2 

-32.08 

-17.19 

Navy 

F 

Production 

2 

-4.55 

-24.14 

Navy 

Cost  Reimbursable 

Development 

2 

-17.83 

-6.91 

Amy 

Cost  Reimbursable 

Production 

2 

-8.33 

-20.00 

Amy 

Fixed  Price 

Production 

2 

0.00 

-25.64 

Air  Force 

Fixed  Price 

Development 

2 

4.17 

-9.40 

Air  Force 

Cost  Reimbursable 

Development 

2 

0.96 

-1.93 

Navy 

Fixed  Price 

Production 

2 

-0.53 

-5.62 

Navy 

Fixed  Price 

Production 

2 

-2.45 

-2.40 

Air  Force 

Cost  Reimbursable 

Production 

2 

1.65 

-4.09 

Air  Force 

Fixed  Price 

Production 

2 

0.51 

10.04 

Air  Force 

Cost  Reimbursable 

Development 

2 

4.55 

i 

Air  Force 

Cost  Reimbursable 

Production 

2 

Air  Force 

Fixed  Price 

Production 

2 

3.64 

-3.51 

Air  Force 

Cost  Reimbursable 

Development 

2 

-4.58 

-3.04 

Air  Force 

Fixed  Price 

Production 

2 

5.48 

1.90 

Air  Force 

Fixed  Price 

Production 

2 

0.36 

1.37 

Air  Force 

Fixed  Price 

Production 

2 

-10.05 

-3.76 

Navy 

Fixed  Price 

Production 

2 

5.72 

-16.56 

Air  Force 

Fixed  Price 

Development 

2 

-6.38 

-12.62 
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msmmmm 

Fixed  Price 
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Mavy 

Fixed  Price 

Production 

2 

0.00 

-17.14 

Navy 

Fixed  Price 

Production 

2 

-19.30 

-4.27 

Navy 

Cost  Reimbursable 

Development 

2 

-0.72 

-1.25 

Army 

Fixed  Price 

Production 

2 

8.86 

-5.20 

Amy 

Fixed  Price 

Production 

2 

-11.97 

-18.86 

Amy 

Fixed  Price 

Production 

2 

-12.15 

-17.99 

Navy 

Cost  Reimbursable 

Production 

2 

-42.86 

-22.22 

Amy 

Cost  Reimbursable 

Development 

2 

17.14 

0.00 

Amy 

Cost  Reimbursable 

Development 

2 

-5.42 

-10.80 

Navy 

Cost  Reimbursable 

Development 

2 

0.00 

-10.71 

Navy 

Cost  Rein^ursable 

Development 

2 

-6.90 

-1.91 

Navy 

Cost  Reimbursable 

Development 

2 

-13.79 

-9.38 

Navy 

Cost  Reimbursable 

Development 

2 

-4.55 

-10.71 

Navy 

Cost  Reimbursable 

Development 

2 

-7.69 

0.00 

Navy 

Cost  Reimbursable 

Development 

2 

-14.51 

-23.00 

Air  Force 

Fixed  Price 

Production 

2 

3.51 

-0.16 

Air  Force 

Fixed  Price 

Production 

2 

-40.97 

-31.93 

Air  Force 

Fixed  Price 

Development 

2 

-9.94 

-9.04 

Navy 

Fixed  Price 

Development 

2 

-31.83 

-25.17 

Navy 

Fixed  Price 

Production 

2 

-3.32 

-9.05 

Navy 

Cost  Reimbursable 

Development 

2 

-11.53 

-1.96 

Navy 

Fixed  Price 

Production 

2 

-10.81 

-7.68 

Navy 

Fixed  Price 

Production 

2 

-13.95 

-15.60 

Navy 

Fixed  Price 

Production 

2 

-12.50 

-5.88 

Navy 

Fixed  Price 

Production 

2 

-2.97 

22.69 

Army 

Fixed  Price 

Development 

2 

8.49 

-2.22 

Amy 

Fixed  Price 

Production 

2 

-2.04 

-5.07 

Am^ 

Fixed  Price 

Production 

2 

2.38 

-2.33 

Amy 

Fixed  Price 

Production 

2 

0.00 

-12.50 

Army 

Cost  Reimbursable 

Development 

2 

2.78 

-5.87 

Navy 

Fixed  Price 

Production 

2 

1.96 

-3.34 

Navy 

Fixed  Price 

Production 

2 

-16.32 

4.65 

Navy 

Fixed  Price 

Production 

2 

-3.78 

-7.24 

Navy 

Fixed  Price 

Production 

2 

-3.70 

-21.01 

Navy 

Fixed  Price 

Production 

2 

4.88 

0.00 

Navy 

Cost  Reimbursable 

Development 

2 

-16.67 

-33.33 

Navy 

Fixed  Price 

Production 

2 

-10.34 

0.00 

Navy 

Fixed  Price 

Production 

2 

7.50 

-10.79 

Navy 

Fixed  Price 

Production 

2 

-3.90 

-4.01 

Navy 

Cost  Reindsursable 

Production 

2 

-2.12 

-5.79 

Navy 

Cost  Reimbursable 

Production 

2 

-0.65 

-4.97 

Amy 

Fixed  Price 

Production 

2 

-1.09 

-11.54 

Amy 

Fixed  Price 

Production 

2 

-15.52 

-46.30 

Amv 

Fixed  Price 

Production 

2 

-41.24 

-27.93 

Amy 

Fixed  Price 

Production 

2 

-8.88 

-9.05 

Amy 

Fixed  Price 

Production 

2 

-7.14 

-6.67 

Amy 

Fixed  Price 

Production 

2 

-8.62 

-33.30 

Amy 

Fixed  Price 

Development 

2 

11.11 

0.00 

Amy 

Cost  Reimbursable 

Devel^ment 

2 

8.33 

Amy 

Cost  Reimbursable 

Development 

2 

-16.82 

-12.16 

Amy 

Fixed  Price 

Production 

2 

-2.81 

-47.95 

Amy 

Cost  Reimbursable 

Production 

2 

9.24 

-13.85 

Amy 

Cost  Reimbursable 

Development 

2 

6.25 

-5.88 

Amy 

Fixed  Price 

Production 

2 

3.75 

-27.93 

Amy 

Fixed  Price 

Production 

2 

-19.71 

-42.54 
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Azny 

Fixed  Price 

Production 

- 2 - 

-25.35 

-31.55 

hroY 

Fixed  Price 

Production 

2 

1.97 

-20.09 

Azny 

Fixed  Price 

Production 

2 

-10.80 

-55.00 

Azny 

Fixed  Price 

Production 

2 

-9.13 

-8.58 

Kattjf 

Cost  Reimbursable 

Development 

2 

0.00 

0.00 

Navy 

Fixed  Price 

Production 

2  • 

14.81 

-0.61 

Navy 

Fixed  Price 

Production 

2 

14.75 

4.06 

Navy 

Fixed  Price 

Production 

2 

7.63 

13.41 

Navy 

Fixed  Price 

Production 

2 

-2.87 

-0.83 

Navy 

Fixed  Price 

Production 

2 

1.30 

-2.43 

Navy 

Fixed  Price 

Production 

2 

0.49 

-7.01 

Navy 

Fixed  Price 

Production 

2 

2.88 

-0.95 

Navy 

Fixed  Price 

Production 

2 

9.19 

-8.32 

Navy 

Fixed  Price 

Production 

2 

-5.97 

9.84 

Air  Force 

Fixed  Price 

Production 

2 

-11.16 

-14.11 

Air  Force 

Fixed  Price 

Development 

2 

-7.28 

-23.13 

Air  Force 

Fixed  Price 

Production 

2 

1.88 

1.34 

Air  Force 

Fixed  Price 

Production 

2 

4.11 

-8.79 

Air  Force 

Fixed  Price 

Production 

2 

3.20 

21.36 

Air  Force 

Fixed  Price 

Production 

2 

-4.18 

Air  Force 

Fixed  Price 

Development 

2 

2.31 

-0.20 

Air  Force 

Fixed  Price 

Development 

2 

1.84 

-3.06 

Air  Force 

Fixed  Price 

Development 

2 

4.14 

-2.79 

Air  Force 

Fixed  Price 

Production 

2 

1.84 

-2.40 

Air  Force 

Fixed  Price 

Production 

2 

0.00 

-11.15 

Air  Force 

Fixed  Price 

Production 

2 

0.75 

-9.97 

Air  Force 

Fixed  Price 

Production 

2 

3.08 

-5.43 

Navy 

Fixed  Price 

Production 

2 

2.26 

-3.61 

Navy 

Coat  Reimbursable 

Development 

2 

-9.83 

-10.09 

Navy 

Fixed  Pric«9 

Production 

2 

0.68 

3.27 

Navy 

Cost  Reimbursable 

Development 

2 

-0.09 

-10.52 

Navy 

Cost  Reimbursable 

Production 

2 

-4.43 

-5.19 

Navy 

Cost  Reimbursable 

Production 

2 

-2.27 

-15.38 

Navy 

Coat  Reimbursable 

Production 

2 

-0.73 

-14.02 

Navy 

Cost  Reimbursable 

Production 

2 

3.19 

-11.33 

Navy 

Cost  Reimbursable 

Production 

2 

-2.42 

-4.93 

Navy 

Fixed  Price 

Production 

2 

-2.27 

46.67 

Navy 

Cost  Reimbursable 

Development 

2 

0.00 

0.00 

Navy 

Cost  Reimbursable 

Development 

2 

3.50 

-1.03 

Navy 

Fixed  Price 

Production 

2 

0.24 

-7.21 

Navy 

Fixed  Price 

Production 

2 

9.19 

-8.32 

Air  Force 

Fixed  Price 

Development 

2 

-24.29 

Fixed  Price 

Production 

2 

0.00 

-17.65 

Air  Force 

Fixed  Price 

Production 

2 

0.00 

-21.41 

Air  Force 

Fixed  Price 

Production 

2 

10.90 

-36.63 

Air  Force 

Fixed  Price 

Production 

2 

5.26 

-17.99 

Fixed  Price 

Production 

2 

2.15 

-19.98 

Air  Force 

Fixed  Price 

Production 

2 

0.80 

-7.98 

Arsoy 

Cost  Reimbursable 

Development 

2 

-16.16 

-12.24 

Navy 

Coat  Reimbursable 

Development 

2 

-7.56 

-5.03 

Navy 

Fixed  Price 

Production 

2 

-27.32 

Navy 

Fixed  Price 

Production 

2 

-9.89 

-22.84 

Navy 

Fixed  Price 

Production 

2 

0.00 

-15.69 

Air  Force 

Fixed  Price 

Development 

2 

-26.67 

-11.76 

Navy 

Fixed  Price 

Production 

2 

6.70 

0.19 

Navy 

Cost  Reimbursable 

Development 

2 

0.00 

-1.66 
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Fixed  Price 

Development 

2 

1 

00 

5 

Fixed  Price 

Production 

2 

-12.56 

-30.69 

Fixed  Price 

Production 

2 

-48.55 

-37.07 

Fixed  Price 

Development 

2 

6.84 

-22.03 

Fixed  Price 

Production 

2 

1.85 

0.00 

Fixed  Price 

Production 

2 

-10.00 

Fixed  Price 

Production 

2 

-0.60 

-1.33 

Fixed  Price 

Development 

2 

-1.29 

-11.19 

Fixed  Price 

Development 

2 

2.92 

-17.42 

Cost  Reimbursable 

Development 

2 

-10.00 

-16.67 

Fixed  Price 

Production 

2 

-44.24 

6.70 

Fixed  Price 

Production 

2 

-9.35 

45.68 

Fixed  Price 

Production 

2 

2.56 

6.11 

Fixed  Price 

Production 

2 

-2.19 

5.45 

Fixed  Price 

Production 

2 

9.13 

-21.32 

Fixed  Price 

Production 

2 

4.04 

-5.20 

Fixed  Price 

Production 

2 

10.00 

-11.91 

Fixed  Price 

Production 

2 

14.76 

-19.31 

Fixed  Price 

Production 

2 

-2.70 

-5.13 

Fixed  Price 

Production 

2 

3.56 

-23.57 

Fixed  Price 

Production 

2 

3.30 

-10.00 

Fixed  Price 

Production 

2 

7.00 

-5.90 

Fixed  Price 

Production 

2 

3.21 

-1.14 

Fixed  Price 

Production 

2 

2.89 

11.64 

Fixed  Price 

Production 

2 

4.87 

7.56 

Fixed  Price 

Production 

2 

3.59 

0.41 

Fixed  Price 

Production 

2 

-2.63 

4.64 

Fixed  Price 

Production 

2 

7.86 

-11.30 

Fixed  Price 

Production 

2 

0.00 

-4.88 

Fixed  Price 

Production 

2 

1.52 

-4.48 

Fixed  Price 

Production 

2 

9.94 

-26.41 

Fixed  Price 

Production 

2 

40.00 

-16.67 

Fixed  Price 

Production 

2 

0.00 

-10.59 

Fixed  Price 

Production 

2 

12.13 

-44.86 

Fixed  Price 

Production 

2 

4.27 

-26.22 

Fixed  Price 

Development 

2 

-4.17 

-3.85 

Cost  ReiInburs^d>le 

Development 

2 

1.06 

-4.52 

Fixed  Price 

Development 

2 

-2.41 

-5.17 

Cost  Reimbursable 

Development 

2 

-1.89 

-3.75 

Fixed  Price 

Production 

2 

1.43 

-17.95 

Fixed  Price 

Production 

2 

11.91 

-23.11 

Fixed  Price 

Development 

2 

-7.38 

-9.88 

Cost  Reimbursable 

Development 

2 

1.48 

-7.11 

Fixed  Price 

Development 

2 

0.64 

-7.96 

Fixed  Price 

Development 

2 

9.52 

-5.13 

Fixed  Price 

Development 

2 

3.00 

-9.05 

Fixed  Price 

Development 

2 

-4.01 

-7.18 

Fixed  Price 

Development 

2 

-3.24 

-5.41 

Fixed  Price 

Development 

2 

-0.22 

-6.35 

Fixed  Price* 

Development 

2 

12.14 

-17.16 

Fixed  Price 

Development 

2 

-7.14 

Fixed  Price 

Development 

2 

-2.30 

-7.22 

Fixed  Price 

Production 

2 

0.00 

0.00 

Fixed  Price 

Production 

2 

-7.69 

-27.78 

Fixed  Price 

Production 

2 

2.65 

5.61 

Fixed  Price 

Production 

2 

-5.66 

-6.77 
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Air  Force 
Air  Force 
Air  Force 


Force 


Force 

Force 

Force 

Force 

Force 

Force 

Force 

Force 


Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Cost  Reimbursed)  le 
Fixed  Price 
Fixed  Price 
Cost  Reinbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Coat  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Coat  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Cost  Reimbursed)le 
Cost  Reimbursable 
Cost  Reimbursable 


Development 

Development 

Development 

Development 

Development 

Development 

Development 

Development 

Development 

Development 

Development 

Development 

Development 

Production 

Development 

Development 

Development 

Production 

Production 

Production 

Production 

Development 

Development 

Development 

Production 

Production 

Production 

Production 

Production 

Production 

Development 

Production 

production 

Production 

Production 

Production 

Production 

Production 

Development 

Development 

Production 

Production 

Development 

Development 

Production 

Production 

Production 

Production 

Development 

Development 

Development 

Production 

Production 

Development 

Development 

Production 


2 

-11.80 

2 

-9.91 

2 

1.85 

2 

0.00 

2 

-50.00 

2 

-19.72 

2 

-10.96 

2 

-3.25 

2 

0.36 

2 

-2.49 

2 

9.02 

2 

0.07 

2 

3.65 

2 

4.50 

2 

-30.77 

2 

-14.26 

2 

-1.40 

2 

5.71 

2 

-5.91 

2 

0.00 

2 

2.50 

2 

-8.43 

3 

-2.80 

3 

-0.23 

3 

0.90 

3 

0.56 

3 

-1.06 

3 

0.92 

3 

-2.34 

3 

8.42 

3 

1.88 

3 

12.46 

3 

0.60 

3 

0.25 

3 

-1.99 

3 

4.91 

3 

-8.96 

3 

-2.13 

3 

-10.91 

3 

4.73 

3 

0.00 

3 

1.96 

3 

-47.92 

3 

12.69 

3 

-20.00 

3 

0.00 

3 

7.14 

3 

-4.55 

3 

-1.96 

3 

-4.90 

3 

-36.28 

3 

0.00 

3 

0.71 

3 

-15.79 

3 

-2.80 

-20.00 

-11.95 

-1.32 

0.00 

0.00 

-4.19 

-5.00 

-9.39 

-1.00 

-5.06 

-18.75 

-17.70 

-22.23 

-28.57 

1.43 

0.00 

-11.11 

-0.06 

-2.19 

-4.09j 

-4.31 

-1.73 

-2.62 

1.05 

-7.61 

-8.29 

-8.61 

-9.57 

-13.44 

-12.63 

-4.75 

-7.22 

-7.59 

-1.29 

-7.69 

-5.67 

0.00 

-6.61 

-8.68 

-16.40 

-43.18 

-24.00 

-12.50 

-1.49 

-1.88 

-7.55 

-8.21 

-7.94 

-3.17 

-2.56 

-18.77 
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AmY 

Fixed  Price 

Production 

- 3 - 

Air  Force 

Fixed  Price 

Development 

3 

-3.85 

-7.45 

Air  Force 

Cost  ReindSursable 

Development 

3 

1.19 

-0.70 

Navy 

Fixed  Price 

Production 

3 

-1.15 

-2.96 

Air  Force 

Cost  Reimbursable 

Production 

3 

-4.96 

-8.24 

Air  Force 

Fixed  Price 

Production 

3 

-12.84 

6.91 

Air  Force 

Fixed  Price 

Production 

3 

-5.49 

-5.20 

Air  Force 

Fixed  Price 

Production 

3 

-18.07 

-21.84 

Air  Force 

Cost  Reimbursable 

Development 

3 

15.24 

0.00 

Air  Force 

Fixed  Price 

Production 

3 

-1.63 

-1.17 

Air  Force 

Coat  Reimbursable 

Development 

3 

-1.67 

-1.68 

Air  Force 

Fixed  Price 

Production 

3 

2.62 

Air  Force 

Fixed  Price 

Production 

3 

-0.28 

0.53 

Air  Force 

Fixed  Price 

Production 

3 

-7.88 

-7.54 

Air  Force 

Fixed  Price 

Production 

3 

-3.46 

-1.55 

Navy 

Fixed  Price 

Production 

3 

-7.06 

-22.02 

Air  Force 

Fixed  Price 

Development 

3 

-5.42 

-7.53 

Air  Force 

Fixed  Price 

Production 

3 

-2.21 

-8.13 

Navy 

Fixed  Price 

Production 

3 

1.39 

0.00 

Navy 

Fixed  Price 

Production 

3 

-3.98 

-6.40 

Navy 

Fixed  Price 

Production 

3 

8.05 

-3.25 

Navy 

Fixed  Price 

Production 

3 

-25.18 

-3.32 

Navy 

Cost  Reimbursable 

Development 

3 

-0.67 

-3.41 

Amy 

Fixed  Price 

Production 

3 

8.52 

-8.23 

Amy 

Fixed  Price 

Production 

3 

-8.54 

-17.27 

Amy 

Fixed  Price 

Production 

3 

-21.22 

-15.56 

Navy 

Cost  Reimbursable 

Production 

3 

-50.95 

-17.91 

Amy 

Cost  Reimbursable 

Development 

3 

20.13 

-2.78 

Amy 

Cost  Reimbursable 

Development 

3 

-14.94 

-8.02 

Navy 

Cost  Reimbursable 

Development 

3 

-12.12 

-0.76 

Navy 

Cost  Reimbursable 

Development 

3 

-15.38 

-7.14 

Navy 

Cost  Reimbursable 

Development 

3 

0.00 

-20.00 

Navy 

Cost  Reimbursable 

Develqpment 

3 

-8.39 

0.00 

Navy 

Cost  Reimbursable 

Development 

3 

-19.30 

-12.15 

Air  Force 

Fixed  Price 

Production 

3 

-1.27 

1.27 

Air  Force 

Fixed  Price 

Production 

3 

-51.32 

-32.14 

Air  Force 

Fixed  Price 

Development 

3 

-16.10 

-6.47 

Navy 

Fixed  Price 

Development 

3 

-37.41 

-22.71 

Navy 

Fixed  Price 

Production 

3 

-0.35 

-6.21 

Amy 

Cost  Reimbursable 

Development 

3 

0.34 

-2.49 

Navy 

Cost  Reimbursable 

Development 

3 

-7.73 

-10.77 

Navy 

Fixed  Price 

Production 

3 

-18.15 

-8.55 

Navy 

Fixed  Price 

Production 

3 

-21.78 

-9.18 

Navy 

Fixed  Price 

Production 

3 

-1.15 

0.00 

Navy 

Fixed  Price 

Production 

3 

-4.06 

-3.47 

Navy 

Fixed  Price 

Production 

3 

-7.41 

-3.57 

Navy 

Fixed  Price 

Production 

3 

-1.84 

-6.01 

Navy 

Fixed  Price 

Production 

3 

-12.39 

-3.28 

Amy 

Fixed  Price 

Development 

3 

7.87 

-9.91 

Axay 

Fixed  Price 

Production 

3 

-2.33 

-7.02 

Amy 

Fixed  Price 

Production 

3 

0.75 

-4.66 

Axny 

Fixed  Price 

Production 

3 

-13.33 

-11.69 

Amy 

Cost  Reisiburaable 

Devel^ment 

3 

-5.77 

-4.48 

Navy 

Fixed  Price 

Production 

3 

4.05 

-2.94 

Navy 

Fixed  Price 

Production 

3 

-17.76 

4.89 

Navy 

Fixed  Price 

'Production 

3 

-9.90 

-8.24 
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Amy 

Army 

Azny 

Amy 

Army 


Azny 

Azny 

Azny 

Azny 

Azny 

Azny 

Azny 

Amy 

Azny 

Azny 

Army 

Azny 

Azny 

Azny 


osz  ReziDDurs 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Coat  Reimbursable 
Fixed  Price 
Cost  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Coat  Reimbursable 


Development 

Production 

Production 

Development 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Development 

Development 

Development 

Production 

Production 

Development 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Development 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Production 

Development 

Production 

Production 

Production 

Production 

Production 

Development 

Development 

Development 

Development 

Production 

Production 

Production 

Production 

Production 

Production 

Development 


-5.66 

6.95 

-8.57 


-3. 

30 

-2 

.66 

-6. 

69 

-11 

.11 

-24 

.31 

-45 

35 

-10 

.42 

-4 

.55 

-27 

.66 

0 

.00 

2 

.31 

-17 

.83 

0 

.40 

-1 

.98 

3 

.25 

-58 

.95 

-2 

.54 

-22 

09 

-4 

.27 

-10 

.57 

-3 

.33 

-17 

.35 

-2 

.48 

14 

.91 

-12 

.56 

16 

39 

5 

09 

-5. 

94 

0 

.85 

1 

.68 

4 

.34 

8 

.56 

-6 

98 

-8 

57 

-15 

.06 

1 

.03 

2 

.23 

1 

50 

-2. 

95 

2 

.96 

2 

.99 

2 

.58 

2 

.21 

3.47 
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Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Fixed  Price 
Cost  Reimbursable 
Coat  Reimbursable 
Fixed  Price 
Coat  Reimbursable 
Cost  Reimbursable 
Cost  Reimbursable 


-0.76 

-0.89 

6.35 

-2.21 

-0.59 

-1.25 

-6.47 

-1.36 

-2.59 

-2.54 

-33.07 

-4.75 

-11.60 

-4.65 

-0.88 

-6.04, 

-40.11 

-10.02 

-19.00 

-8.20 

3.21 

-2.81 

-5.58 

-7.71 

-2.51 

-5.94 

6.36 

-4.34 

-75.88 

-3.86 

-2.26 

0.00 

-0.77 

-4.23 

-18.25 

-1.56 

B-18 


Appendix  C 


Cost  Variance  SAS  Programs  and  SAS  Outputs 


options  linesize  ■ 

•  options  pages! ze  -  26; 


data  thesis; 

input  type  phase 
%include  buclcdon; 


service  quarter  cv  sv, 


proc  gl®;  .  _ 

class  phase  type  quarter 


service; 


model  cv 


phase 

type (phase) 
quarter (type  phase) 
service (quarter  type 


phase) ; 


means  phase 

type (phase) 

quarter (type  phase) 

service (quarter  type  phase). 

otitout  out-check  p-cvhat  student-sresid; 


proc  plot  data*checks 
plot  3resicl*cvhat; 

proc  univariate  data-check  plot  normal 
var  sresid; 
run; 


C-1 


The  SAS  System  1 

09:39  Monday,  June  20,  1994 

General  Linear  Models  Procedure 
Class  Level  Information 


Class 

Levels 

Values 

PHASE 

2 

1  2 

TYPE 

2 

1  2 

QOARTER 

4 

12  3  4 

SERVICE 

3 

12  3 

Number  of  observations  in  data  set  <■  969 


C-2 


The  SAS  System 


09:39  Monday,  June  20,  1994 


General 

Linear  Models 

Procedure 

i:  CV 

Sum  of 

Mean 

DF 

Squares 

Square 

F  Value 

Pr  >  F 

46 

18203.66065 

395.73175 

2.34 

0.0001 

922 

156144.44308 

169.35406 

968 

174348.10373 

R-Square 

C.V. 

Root  MSE 

CV  Mean 

0.104410 

-314.6066 

13.01361 

-4.136471 

^SE*TYPE) 

ryPE»Q0AR) 


DF 

Type  ITI  SS 

Mean  Square 

F  Value 

Pr  >  F 

1 

309.677037 

309.677037 

1.83 

0.1/66 

2 

688.136306 

344.068153 

2.03 

0.1317 

12 

9948.103437 

829.008620 

4.90 

0.0001 

31 

5934.360860 

191.430995 

1.13 

0.2866 

C-3 


Th«  SAS  System 


5 

09:39  Monday,  June  20,  1994 


Plot  of  SRESID*CVHAT.  Legend:  A  ••  1  obs,  B  •  2  obs,  etc. 


SRESIO 


4  + 


A 
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A 

A 

A 
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CA  B 

B 

A 

A 

A 

A 

A  A 

BB 

A 

A 

E 

CB 
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A 

D 

A 

A 

H 

M  CJAFCDH 

FC 

B 

D  B 

C 

A 

D 

UAAPAWMLX 

NC 

A 

F  C 

F 

A 

G  CLAOJQZ 

ZN 

C 

PBC 

F 

C 

BA 

E 

HILV 

ZC 

J 

BZAE 

J 

B 

F 

lAIJGQ 

HE 

F 

lEJ 

M 

E 

B 

C 

DFEI 

GB 

C 

AAD 

C 

A 

A 

CA 

D 

DEED 

CB 

C  B 

B 

C 

B 

0 

BACE 

DC 

A 

A 

B 

B 

C 

BAAC 

AB 

AA 

A 

B 

A 

C 

A 

A 

A 

A 

A 

A 

C 

B 

A 

A 

A 

B 

A 

B 

A 

A 

A  B 

AA 

A 

B 

A 

A  AA 

BA 

A 

A 

A 

A 

A  A 
A 

A 


A 


A 


B 


-6  + 


A 


I  A 

I 

-8  + 

I 

- 4 - 4- - + - 4 - +, 

-20  -10  0  10  20 


CVHAT 

MOTE:  2  obs  had  missing  values.  31  obs  bidden. 
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The  SAS  System  6 

09:39  Monday,  June  20,  1994 

Univariate  Procedure 

Variable-SRESID 


Moments 


N 

967 

Sum  Wgts 

967 

Mean 

0 

Sum 

0 

Std  Dev 

1.010013 

Variance 

1.020126 

Skewness 

-1.81509 

Kurtosis 

7.948211 

uss 

985.442 

CSS 

985.442 

cv 

Std  Mean 

0.03248 

T: Mean-0 

0 

Pr>|Tl 

1.0000 

Hum  0 

967 

Mum  >  0 

581 

M(Sign) 

97.5 

Pr>-|M1 

0.0001 

Sgn  Rank 

39633 

Pr>-|S| 

0.0001 

H:  Normal 

0.868986 

Pr<W 

0.0001 

Quantiles (Def-5) 


100%  Max 

3.748137 

99%. 

2.016404 

75%  Q3 

0.511385 

95% 

1.151745 

50%  Med 

0.153875 

90% 

0.916663 

25%  Q1 

-0.30439 

10% 

-1.05064 

0%  Min 

-7.46235 

5% 

-1.69244 

1% 

-3.76635 

Range 

11.21049 

Q3-Q1 

0.815772 

Mode 

-0.10645 

Extremes 


Lowest 

Obs 

Highest 

Obs 

-7.46235( 

54) 

2.614778( 

621) 

-5. 70846 < 

11) 

2.720679( 

30) 

-5.25297 ( 

966) 

2. 994102 ( 

363) 

-4. 96597 ( 

938) 

3.277905( 

904) 

-4. 64623 ( 

655) 

3.748137( 

120) 

Missing  Value 

Count  2 

%  Count/Nobs  0.21 
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The  SAS  System 
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09:39  Monday,  June  20,  1994 


Univariate  Procedure 


Variable-SRESID 


Histogram  f 

3.75+*  1 

.*  1 

.*  5 

.*  4 

.***  15 

.****•**•  52 

*«***4k*******************  172 

,*********•*•**•***••*•**«**••  202 

,*««******••*  02 

_******  42 

-1.75+***  18 

.**  13 

.  **  9 

,  **  8 

.*  5 

.*  1 

.*  3 

.*  1 

.*  1 

-7.25+*  1 

- 4. - + - + - +— — + - + - + - + - 4 - 

*  may  represent  up  to  7  counts 


Boxplot 


0 

0 

0 

I 

4 - 4 

•  __4  —  * 
4_ - 4 


0 

0 

0 

• 

* 

* 

* 


* 
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The  SAS  System 
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09:39  Monday,  June  20,  1994 


Univariate  Procedure 


Variable-SRESID 


3.75+ 


Normal  Probability  Plot 


+++*» 

++++***»* 


++++»' 

********** 
*******4^ 

*»**++++ 

***+++ 

•1.75+  ++**+ 

+++++*** 

1+  ■ 

*•* 

•  * 

I* 

I* 

I  * 


-7.25+» 

+ - + - + - + - + - + - + - + - + - + - + 


-2  -1 


+1 


+2 
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options  lineaize  »  78; 

*  options  pagesize  -  26; 


data  thesis ; 

input  type  phase  service  quarter  cv  sv; 
%include  buckdon; 

proc  glm; 

class  type  service  quarter  phase; 

model  cv  -  type 

service (type) 
quarter (service  type) 
phase (quarter  service  type) 

run; 


C-8 


The  SAS  System 


09:40  Monday,  June  20 


1 

1994 


General  Linear  Models  Procedure 
Class  Level  Information 


Class 

Levels 

Values 

TYPE 

2 

1 

2 

SERVICE 

3 

1 

2  3 

QUARTER 

4 

1 

2  3  4 

PHASE 

2 

1 

2 

of  observations 

in  data  set 

969 


The  SAS  System  2 

09:40  Monday,  June  20,  1994 


General 

Linear  Models 

Procedure 

Dependent  Variable: 

CV 

Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value  Pr  >  F 

Model 

46 

18203.66065 

395.73175 

2.34  0.0001 

Error 

922 

156144.44308 

169.35406 

Corrected  Total 

968 

174348.10373 

R- 

■Square 

C.V. 

Root  MSE 

CV  Mean 

0. 

104410 

-314.6066 

13.01361 

-4.136471 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

TYPE 

1 

425.363211 

425.363211 

2.51 

0.1133 

SERVICE (TYPE) 

4 

544.405475 

136.101369 

0.80 

0.5229 

QOARTE (TYPE*SERVICE) 

18 

8713.911259 

484.106181 

2.86 

0.0001 

PEAS (TYPE*SERV*QOAR) 

23 

4380.110461 

190.439585 

1.12 

0.3106 

C-9 


options  linesize  -  78; 

*  options  pagesize  ••  26; 

data  thesis; 

input  type  phase  service  cpiarter  cv  sv; 
%include  buckdon; 

proc  glm; 

class  phase  service  quarter  type; 

model  cv  >  phase 

service (phase) 
quarter (service  phase) 
type (quarter  service  phase) 


run; 
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The  SAS  System  1 

09:41  Monday,  ~jne  20,  1994 

General  Linear  Models  Procedure 
Class  Level  Infonaation 


Class 

Levels 

Values 

PHASE 

2 

1 

2 

SERVICE 

3 

1 

2  3 

QUARTER 

4 

1 

2  3  4 

TTPE 

2 

1 

2 

Number  of  observations  in  data  set  •  969 


The  SAS  System  2 

09:41  Monday,  June  20,  1994 


General 

Linear  Models 

procedure 

Dependent 

Variable :  CV 

Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Pr  >  F 

Model 

46 

18203. 6606S 

395.73175 

2.34 

0.0001 

Error 

922 

156144.44308 

169.35406 

Corrected 

Total  968 

174348.10373 

R-Square 

c.v. 

Root  MSE 

CV  Mean 

0.104410 

-314.6066 

13.01361 

-4.136471 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

PHASE 

1 

175.130787 

175.130787 

1.03 

0.3095 

SERVICE (PHASE) 

4 

916.420171 

229.105043 

1.35 

0.2485 

QOARTE (PHASE*SERVIC) 

18 

5634.590389 

313.032799 

1.85 

0.0168 

TYPE  <PHAS*SERV*QOAR) 

23 

4892.459283 

212.715621 

1.26 

0.1881 

C-11 


J^ppendix  D:  Sch«dul«  Variance 
SAS  Programs  and  SAS  Outputs 


options  linesize  -  78; 

*  options  pagesize  ••  26; 

data  thesis; 

input  type  phase  service  quarter  cv  sv 
%include  buclcdon; 

proc  glm; 

class  phase  type  quarter  service; 

model  sv  »  phase 

type (phase) 

quarter (type  phase) 

service (quarter  type  phase) ; 


means  phase 

type (phase) 

quarter (type  phase) 

service (quarter  type  phase); 

output  out-check  p-svhat  student-sresid; 

proc  plot  data-check; 
plot  sresid*svhat; 

proc  univariate  data-check  plot  normal; 
var  sresid; 
run; 


D-1 


The  SAS  System  1 

09:38  Monday,  June  20,  1994 

General  Linear  Models  Procedure 
Class  Level  Information 


Class 

Levels 

Values 

PHASE 

2 

1 

2 

TYPE 

2 

1 

2 

QUARTER 

4 

1 

2  3  4 

SERVICE 

3 

1 

2  3 

Number  of  observations  in  data  set  -  969 


D-2 


Th«  SAS  Systen 

09:38  Monday,  June 


General 

Linear  Models 

Procedure 

Dependent  Variable :  SV 

Source  OF 

Sum  of 
Squares 

Mean 

Square 

F  Value 

Model 

46 

22869.39862 

497.16084 

4.32 

Error 

922 

106169.86957 

115.15170 

Corrected  Total 

968 

129039.26820 

R-Square 

C.V. 

Root  MSE 

0.177228 

-166.4754 

10.73088 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

PHASE 

1 

65.85931 

65.85931 

0.57 

TYPE (PHASE) 

2 

118.47127 

59.23563 

0.51 

QUARTER (PHASE*TYPE) 

12 

5895.80977 

491.31748 

4.27 

SERV (PHAS*TYPE*QOAR) 

31 

17271.74601 

557.15310 

4.84 

D-3 


2 

20,  1994 


Pr  >  F 
0.0001 


SV  Mean 
6.445924 


Pr  >  F 

0.4497 

0.5980 

0.0001 

0.0001 


The  SAS  System  ^ 

09:38  Monday,  June  20,  1994 

Plot  of  SRESID*SVHAT.  Legend:  A  -  1  obs,  B  -  2  obs,  etc. 


SRESID 


A 

A 

AA 

A 

A 

AAA 

A 

A 

AAA 

B 

D 

A 

AB 

A  A  A 

D 

C 

E 

AACAEO 

B  AA  A  A 

C 

F 

D 

BC  LNM 

D  G 

D 

F 

B 

HAGVH 

OIIVBKNZ 

0 

D 

C 

BEADOM 

OLJZCWZZ 

A 

B 

D 

CBEABGK 

IBDNBOJZ 

0 

D 

D 

AAAGFC 

HCFC  EBG 

A 

A 

A 

AB  AGA 

EAAB  B  F 

B 

AC 

BA  C  B 

A 

A 

BBB 

A  B 

B 

A 

A 

A 

A  AB 

A 

AB 

SVHAT 


NOTE:  2  obs  had  missing  values.  128  obs  hidden. 


D-4 


Th«  SAS  System 
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09:38  Monday,  June  20,  1994 


Univariate  Procedure 


Variable-SRESID 


Moments 


N 

967 

Sum  Ngts 

967 

Mean 

0 

Sum 

0 

Std  Dev 

0.99672 

Variance 

0.99345 

Skewness 

1.471944 

Kurtosis 

25.2061 

OSS 

959.673 

CSS 

959.673 

cv 

. 

Std  Mean 

0.032052 

T : Mean>0 

0 

Pr>|T| 

1.0000 

Num  0 

967 

Num  >  0 

538 

M(Sign) 

54.5 

Pr>-|M| 

0.0005 

Sgn  Rank 

16916 

Pr>-)S| 

0.0515 

W :  Normal 

0.85069 

Pr<W 

0.0001 

Quantiles (Def>5) 

100%  Max 

11.90275 

99% 

2.538876 

75%  Q3 

0.36208 

95% 

1.188916 

S0%  Med 

0.058465 

90% 

0.84024 

25%  Q1 

-0.32099 

10% 

-0.97.795 

0%  Min 

-4.79576 

5% 

-1.55868 

1% 

-3.17607 

Range 

16.69851 

Q3-Q1 

0.683075 

Mode 

0.054789 

Extremes 


Lowest 

Obs 

Highest 

Obs 

-4. 79576 ( 

59) 

4. 222568 ( 

76) 

-4.56414  ( 

154) 

4.642425 ( 

343) 

-4. 47046 ( 

58) 

4.73568 ( 

312) 

-3.878( 

111) 

5.558747 ( 

172) 

-3.84291 ( 

54) 

11.90275 ( 

117) 

Missing  Value 

Count  2 

%  Count/Nobs  0.21 
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The  SAS  System 
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09:38  Monday,  June  20,  1994 


Variable-SRESZO 


11. 5+* 


Onivariate  Procedure 


Histogram 


3.5+ 


-4.5+ 


********************************* 


1 

3 

2 

10 

52 

469 

336 

61 

21 

8 

3 


Boxplot 


0 

0 

+ — + — + 

+ - + 

0 

0 

*  ■ 


*  may  represent  up  to  10  counts 

Normal  Probability  Plot 

11.5+ 


3.5+ 


* 

* 


++++* 


I  4.4.******.f4. 

(++****** 

I  *•* 

-4.5+* 

- - 4. - 4. - —4. - 4. - 4 - 4 

-2  -1  0  +1  +2 
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options  linesize  •  78; 

*  *  options  pagesize  -  26; 

data  thesis; 

input  type  phase  service  quarter  cv  sv; 

%include  buclcdon; 

proc  glm; 

class  type  service  quarter  phase; 

model  sv  «  type 

service (type) 
quarter (service  type) 
phase (quarter  service  type) 

run; 


D-7 


The  SAS  System 


1 

09:39  Monday,  June  20,  1994 


General  Linear  Models  Procedure 
Class  Level  Information 


Class 

Levels 

Values 

TTPB 

2 

1  2 

SERVICE 

3 

12  3 

QOARTER 

4 

12  3  4 

PHASE 

2 

1  2 

Number  of  observations  in  data  set  -  969 


The  SAS  System 


09:39  Monday,  June  20,  1994 


• 

General 

Linear  Models 

Procedure 

Dependent  Variable 

:  SV 

Sum  of 

Mean 

Source 

OF 

Squares 

Square 

F  Value 

Pr  >  F 

Model 

46 

22869.39862 

497.16084 

4.32 

0.0001 

Error 

922 

106169.86957 

115.15170 

Corrected  Total 

968 

129039.26820 

R-Square 

C.V. 

Root  MSE 

SV  Mean 

0.177228 

-166.4754 

10.73088 

-6.445924 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

TYPE 

1 

113.207600 

113.207600 

0.98 

0.3217 

SERVICE (TYPE) 

4 

1901.304449 

475.326112 

4.13 

0.0025 

QOARTE (TYPE*SERVICE) 

18 

4465.144698 

248.063594 

2.15 

0.0035 

PEAS (TYPE*SERV*QOAR) 

23 

1476.324084 

64.188004 

0.56 

0.9546 
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options  linesize  -  78; 

*  options  pagesize  -  26; 

data  thesis; 

input  type  phase  service  quarter  cv  sv; 
%include  buckdon; 

proc  glm; 

class  phase  service  cpiarter  type; 

model  sv  -  phase 

service (phase) 
quarter (service  phase) 
type (quarter  service  phase) 


run; 


Tha  SAS  System 


1 

09:40  Monday,  June  20,  1994 


General  Linear  Models  Procedure 
Class  Level  Information 


Class 

Levels 

Values 

PHASE 

2 

1 

2 

SERVICE 

3 

1 

2  3 

QOARTER 

4 

1 

2  3  4 

TYPE 

2 

1 

2 

of  observations 

in  data  set 

969 


The  SAS  System 


2 

09:40  Monday,  June  20,  1994 


General 

Linear  Models 

Procedure 

Dependent  Variable 

:  SV 

Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Pr  >  F 

Model 

46 

22869.39862 

497.16084 

4.32 

0.0001 

Error 

922 

106169.86957 

115.15170 

Corrected  Total 

968 

129039.26820 

R-Square 

C.V. 

Root  MSE 

SV  Mean 

0.177228 

-166.4754 

10.73088 

-6.445924 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

PHASE 

1 

81.942356 

81.942356 

0.71 

0.3991 

SERVICE (PHASE) 

4 

2393.578465 

598.394616 

5.20 

0.0004 

QOARTE (PHASE*SERVIC) 

18 

4088.334459 

227.129692 

1.97 

0.0091 

TYPE (PHAS*SBRV*QOAR) 

23 

3641.672129 

158.333571 

1.37 

0.1122 

D-10 


« 


■» 


options  linesize  •  78; 

*  options  pagesize  •  26; 

data  thesis; 

input  type  phase  service  quarter  cv  sv; 

%include  buckdon; 

proc  glm; 

class  phase  type  quarter  service; 

model  sv  -  phase 

type (phase) 

quarter (type  phase) 

service (quarter  type  phase) ; 

contrast  'Dev,  Cost  Reimb,  Ql' 

service (quarter  type  phase)  1  -1  0, 
service (quarter  type  phase)  1  0  ~1; 

contrast  'Dev,  Cost  Reimb,  Q2' 

service (quarter  type  phase)  0001-10, 
service (quarter  type  phase)  00010-1; 

contrast  'Dev,  Cost  Reimb,  Q3' 

service (quarter  type  phase)  0000001-10, 
service (quarter  type  phase)  00000010  -1; 

contrast  'Dev,  Cost  Reimb,  Q4' 

service (quarter  type  phase)  0000000001-10, 
service (quarter  type  phase)  00000000010-1; 

contrast  'Dev,  Fixed  Price,  Ql' 

service (quarter  type  phase)  000000000000 

1  -1  0, 

service (quarter  type  phase)  000000000000 

1  0  -1; 


contrast  'Dev,  Fixed  Price,  Q2' 

service (quarter  type  phase)  000000000000 

0001-10, 

service (quarter  type  phase)  000000000000 

00010-1; 


contrast  'Dev,  Fixed  Price,  Q3' 

service (quarter  type  phase)  000000000000 

0000001  -1  0, 

service (quarter  type  phase)  000000000000 

00000010-1; 


contrast  'Dev,  Fixed  Price,  Q4' 

service (quarter  type  phase)  000000000000 

0000000001  -1  0, 

service (quarter  type  phase)  000000000000 

00000000010-1; 


contrast  'Prod,  Cost  Reimb,  Ql' 

service (quarter  type  phase)  000000000000 

000000000000 
1  -1; 

contrast  'Prod,  Cost  Reimb,  Q2' 

service (quarter  type  phase)  000000000000 

000000000000 

001-10, 

service (quarter  type  phase)  000000000000 

000000000000 
0010  -1; 
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contrast 


contrast 


contrast 


contrast 


contrast 


contrast 


'Prod,  Cost  Reimb,  Q3' 

service (quarter  type  phase) 


service (quarter  type  phase) 


'Prod,  Cost  Reimb,  Q4' 

service (quarter  type  phase) 


service (quarter  type  phase) 


'Pro,  Fixed  Price,  Ql' 

service (quarter  type  phase) 


service (quarter  type  phase) 


'Pro,  Fixed  Price,  Q2' 

service (quarter  type  phase) 


service (quarter  type  phase) 


'Pro,  Fixed  Price,  Q3' 

service (quarter  type  phase) 


service (quarter  type  phase) 


'Pro,  Fixed  Price,  Q4' 

service (quarter  type  phase) 


service (quarter  type  phase) 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

-1 

c 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

-1 

f 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

-J 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

-1 

c 

', 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

-] 

•  f 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

. 

), 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

L; 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

-] 

0, 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

-] 

L; 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

L 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

- 

I 


run; 


f 
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The  SAS  System 


1 

09:41  Monday,  June  20,  1994 


General  Linear  Models  Procedure 
Class  Iievel  Information 


Class 

Levels 

Values 

PRASE 

2 

1  2 

TYPE 

2 

1  2 

QUARTER 

4 

12  3  4 

SERVICE 

3 

12  3 

Number  of  observations  in  data  set  •  969 


The  SAS  System  2 

09:41  Monday,  June  20,  1994 


General 

Linear  Models 

Procedure 

Dependent  Variable: 

sv 

Sum  of 

Mean 

Source 

OF 

Squares 

Square 

F  Value 

Pr  >  F 

Model 

46 

22869.39862 

497.16084 

4.32 

0.0001 

Error 

922 

106169.86957 

115.15170 

Corrected  Total 

968 

129039.26820 

R- 

-Square 

c.v. 

Root  MSE 

SV  Mean 

0. 

,177228 

-166.4754 

10.73088 

-6.445924 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

P^E 

1 

65.85931 

65.85931 

0.57 

0.4497 

TYPE (PHASE) 

2 

118.47127 

59.23563 

0.51 

0.5980 

QUARTER (PHASE*TYPE) 

12 

5895.80977 

491.31748 

4.27 

0.0001 

SERV (PHAS*TYPE*QOAR) 

31 

17271.74601 

557.15310 

4.84 

0.0001 

Contrast 

DF 

Contrast  SS 

Mean  Square 

F  Value 

Pr  >  F 

Dev,  Cost  Reimb,  Q1 

2 

100.230436 

50.115218 

0.44 

0.6473 

Dev,  Cost  Reimb,  Q2 

2 

130.658553 

65.329277 

0.57 

0.5672 

Dev,  Cost  Reimb,  Q3 

2 

171.897253 

85.948626 

0.75 

0.4744 

Dev,  Cost  Reimb,  Q4 

2 

80.516299 

40.258149 

0.35 

0.7051 

Dev,  Fixed  Price,  Q1 

2 

114.354968 

57.177484 

0.50 

0.6088 

Dev,  Fixed  Price,  Q2 

2 

73.368823 

36.684412 

0.32 

0.7273 

Dev,  Fixed  Price,  Q3 

2 

75.644503 

37.822251 

0.33 

0.7201 

Dev,  Fixed  Price,  Q4 

2 

0.713842 

0.356921 

0.00 

0.9969 

Prod,  Cost  Reimb,  Q1 

1 

12.662862 

12.662862 

0.11 

0.7403 

Prod,  Cost  Reimb,  Q2 

2 

133.407473 

66.703736 

0.58 

0.5605 

Prod,  Cost  Reimb,  Q3 

2 

20.525829 

10.262914 

0.09 

0.9147 

Prod,  Cost  Reimb,  Q4 

2 

14.064525 

7.032263 

0.06 

0.9408 

Pro,  Fixed  Price,  Q1 

2 

7428.654220 

3714.327110 

32.26 

0.0001 

Pro,  Fixed  Price,  Q2 

2 

5826.269202 

2913.134601 

25.30 

0.0001 

Pro,  Fixed  Price,  Q3 

2 

2976.429772 

1488.214886 

12.92 

0.0001 

Pro,  Fixed  Price,  Q4 

2 

112.347453 

56.173727 

0.49 

0.6141 
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options  linesize  -  78; 

*  options  pagesize  -  26; 

data  thesis; 

input  type  phase  service  quarter  cv  sv; 

%include  bur'cdon; 

proc  glm; 

class  phase  type  quarter  service; 
model  sv  -  phase 

type (phase) 

quarter (type  phase) 

service (quarter  type  phase) ; 

contrast  'Dev,  Cost  Reimb,  Ql' 

service (quarter  type  phase)  1  -1  0, 
service (quarter  type  phase)  1  0  -1; 

contrast  'Dev,  Cost  Reimb,  Q2' 

service (quarter  type  phase)  0001-10, 
service (quarter  type  phase)  00010  -1; 

contrast  'Dev,  Cost  Reimb,  Q3' 

service (quarter  type  phase)  0000001-10, 
service (quarter  type  phase)  00000010-1; 

contrast  'Dev,  Cost  Reimb,  Q4' 

service (quarter  type  phase)'  0000000001-10 
service (quarter  type  phase)  00000000010-1 

contrast  'Dev,  Fixed  Price,  Ql' 

service (quarter  type  phase)  000000000000 

1  -1  0, 

service (quarter  type  phase)  000000000000 

1  0  -1; 

contrast  'Dev,  Fixed  Price,  Q2' 

service (quarter  type  phase)  000000000000 

0001-10, 

service (quarter  type  phase)  000000000000 

00010-1; 


contrast  'Dev,  Fixed  Price,  Q3' 

service (quarter  type  phase)  000000000000 

0000001-10, 

service (quarter  type  phase)  000000000000 

00000010-1; 


contrast  'Dev,  Fixed  Price,  Q4' 

service (quarter  type  phase)  000000000000 

0000000001  *1  0 

service (quarter  type  phase)  000000000000 

00000000010  — 1 


contrast  'Prod,  Cost  Reimb,  Ql' 

service (quarter  type  phase)  000000000000 

000000000000 
1  -1; 

contrast  'Prod,  Cost  Reimb,  Q2' 

service (quarter  type  phase)  000000000000 

000000000000 

001-10, 

service (quarter  type  phase)  000000000000 

000000000000 
0  0  10  -1; 
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contrast  'Prod,  Cost  Reimb,  Q3' 

service (quarter  type  phase)  000000000000 
*  000000000000 

000001-10, 

service (quarter  type  phase)  000000000000 

000000000000 

0000010-1; 

contrast  'Prod,  Cost  Reimb,  Q4' 

service (quarter  type  phase)  000000000000 

000000000000 

000000001-10 

service (quarter  type  phase)  000000000000 

000000000000 

0000000010-1 

contrast  'Pro,  Fixed  Price,  Ql' 

service (quarter  type  phase)  000000000000 

000000000000 
00000000000 
1  -1  0, 

service (quarter  type  phase)  000000000000 

000000000000 
00000000000 
1  0  -1; 

estimate  'USAF  vs  USA,  1st  Qtr' 

service (quarter  type  phase)  000000000000 

000000000000 
00000000000 
1  -1  0; 

estimate  'USAF  vs  USN,  1st  Qtr' 

service (quarter  type  phase)  000000000000 

oooooooooooo 
00000000000 
1  0  -1; 

estimate  'USA  vs  USN,  1st  Qtr' 

service (quarter  type  phase)  000000000000 

OOOOOOOOOOOO 
00000000000 
0  1  -1; 

contrast  'Pro,  Fixed  Price,  Q2' 

service (quarter  type  phase)  000000000000 

OOOOOOOOOOOO 

00000000000 

0001-10, 

service (quarter  type  phase)  000000000000 

OOOOOOOOOOOO 

00000000000 

00010-1; 

estimate  'USAF  vs  USA,  2nd  Qtr' 

service (quarter  type  phase)  000000000000 

OOOOOOOOOOOO 

00000000000 

0001-10; 

estimate  'USAF  vs  USN,  2nd  Qtr' 

service (quarter  type  phase)  000000000000 

OOOOOOOOOOOO 
00000000000 
0  0  0  1  0  -1; 

estimate  'USA  vs  USN,  2nd  Qtr' 
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service (quarter  type  phase)  000000000000 

000000000000 

00000000000 

00001-1; 

contrast  'Pro,  Fixed  Price,  Q3' 

service (quarter  type  phase)  000000000000 

000000000000 

00000000000 

0000001-10, 

service (quarter  type  phase)  000000000000 

000000000000 

00000000000 

00000010-1; 

estimate  'USAF  vs  OSA,  3rd  Qtr' 

service (quarter  type  phase)  000000000000 

000000000000 

00000000000 

0000001-10; 

estimate  'USAF  vs  USN,  3rd  Qtr' 

service (quarter  type  phase)  000000000000 

000000000000 

00000000000 

00000010-1; 

estimate  'USA  vs  USK,  3rd  Qtr' 

service (quarter  type  phase)  000000000000 

000000000000 

00000000000 

00000001-1; 

contrast  'Pro,  Fixed  Price,  Q4' 

service (quarter  type  phase)  000000000000 

000000000000 

00000000000 

0000000001-10 

service (quarter  type  phase)  000000000000 

000000000000 
OOOOGOOOOOO 
00000000010  -1 

*  output  out-chec)c  p»cvhat  student»sresid; 

« 

*  proc  plot  data-chec]c; 

*  plot  sresid*cvhat; 

* 

*  proc  univariate  <lata«>check  plot  normal; 

*  var  sresid; 

run; 
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Th«  SAS  System  1 

09:42  Monday,  June  20,  1994 

General  Linear  Models  Procedure 
Class  Level  Information 


Class 

Levels 

Values 

PHASE 

2 

1 

2 

TYPE 

2 

1 

2 

OOARTER 

4 

1 

2  3  4 

SERVICE 

3 

1 

2  3 

Number  of  observations  in  data  set  •  969 
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The  SAS  System 


2 

09:42  Monday,  June  20,  1994 


General  Linear  Models  Procedure 


* 


Dependent  Variable :  SV 


Sum  of 


Mean 


Source 

OF 

Squares 

Square 

F  Value 

Pr  >  F 

Model 

46 

22869.39862 

497.16084 

4.32 

0.0001 

Error 

922 

106169. 869S7 

115.15170 

Corrected  Total 

968 

129039.26820 

R-Square 

c.v. 

Root  MSE 

SV  Mean 

0.177228 

-166.4754 

10.73088 

-6.445924 

Source 

DF 

Type  III  SS 

Mean  Square 

F 

Value 

Pr  >  F 

PHASE 

1 

65.85931 

65.85931 

0.57 

0.4497 

TYPE (PHASE) 

2 

118.47127 

59.23563 

0.51 

0.5980 

QUARTER (PHASE*TYPE) 

12 

5895.80977 

491.31748 

4.27 

0.0001 

SERV (PHAS*TYPE*QOAR) 

31 

17271.74601 

557.15310 

4.84 

0.0001 

Contrast 

DF 

Contrast  SS 

Mean  Square 

f' 

Value 

Pr  >  F 

Dev,  Cost  Reimb, 

Q1 

2 

100.230436 

50.115218 

0.44 

0.6473 

Dev,  Cost  Reimb, 

Q2 

2 

130.658553 

65.329277 

0.57 

0.5672 

Dev,  Cost  Reimb, 

Q3 

2 

171.897253 

85.948626 

0.75 

0.4744 

Dev,  Cost  Reimb, 

Q4 

2 

80.516299 

40.258149 

0.35 

0.7051 

Dev,  Fixed  Price, 

Q1 

2 

114.354968 

57.177484 

0.50 

0.6088 

Dev,  Fixed  Price, 

Q2 

2 

73.368823 

36.684412 

0.32 

0.7273 

Dev,  Fixed  Price, 

Q3 

2 

75.644503 

37.822251 

0.33 

0.7201 

Dev,  Fixed  Price, 

Q4 

2 

0.713842 

0.356921 

0.00 

0.9969 

Prod,  Cost  Reimb, 

Q1 

1 

12.662862 

12.662862 

0.11 

0.7403 

Prod,  Cost  Reimb, 

Q2 

2 

133.407473 

66.703736 

0.58 

0.5605 

Prod,  Cost  Reimb, 

Q3 

2 

20.525829 

10.262914 

0.09 

0.9147 

Prod,  Cost  Reisib, 

Q4 

2 

14.064525 

7.032263 

0.06 

0.9408 

Pro,  Fixed  Price, 

Q1 

2 

7428.654220 

3714.327110 

32.26 

0.0001 

Pro,  Fixed  Price, 

Q2 

2 

5826.269202 

2913.134601 

25.30 

0.0001 

Pro,  Fixed  Price, 

Q3 

2 

2976,429772 

1488.214886 

12.92 

0.0001 

Pro,  Fixed  Price, 

Q4 

2 

112.347453 

56.173727 

0.49 

0.6141 

T  for 

HO:  Pr  > 

|T| 

Std 

Error  of 

Parameter 

Estimate  Parameter-0 

Estimate 

OSAF  vs  USA,  1st 

Qtr 

3 

.0119790 

1.18  0.2368 

2. 

54441471 

USAF  vs  USN,  1st 

Qtr 

-17 

.7150776  -6.89  0.0001 

2. 

57213115 

USA  vs  USM,  1st  Qtr 

-20 

.7270567 

-7.29  0.0001 

2. 

84311610 
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The  SAS  System  3 

09:42  Monday,  June  20,  1994 

General  Linear  Models  Procedure 


Dependent  Variable :  SV 


Parameter 


T  for  HO:  Pr  >  |T| 

Estimate  Parameter-0 


Std  Error  of 
Estimate 


DSAF  vs  OSA,  2nd  Qtr 
USAF  vs  OSH,  2nd  Qtr 
OSA  vs  OSM,.2nd  Qtr 
OSAF  vs  USA,  3rd  Qtr 
USAF  vs  OSN,  3rd  Qtr 
OSA  vs  OSM,  3rd  Qtr 


12.1921969 

-5.3339802 

-17.5261771 

9.4537732 

-2.5875589 

-12.0413321 


5.11 

0.0001 

2.38745581 

2.53 

0.0115 

2.10604140 

7.06 

0.0001 

2.48286272 

4.05 

0.0001 

2.33651317 

1.31 

0.1891 

1.96905096 

4.98 

0.0001 

2.41805913 
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Cost  Variance 


Appendix  B:  Bartlett  Test  Results 
and  Related  Scatter  Plots 


Cat 


(rests  that 

the 

Variance 

a  are  Equal  1 

Level  Count 

Std  Dev 

NeanAbsOif 

to 

Mean  NeanAbsOif 

to  Median 

lAFPFP 

46 

6.40591 

4.911957 

4.911957 

lAPFP 

29 

12.67427 

8.796908 

8.502069 

INPFP 

28 

16.03863 

9.984898 

9.337143 

Test 

F  Ratio 

OF  Nub  of 

Den 

Prob>F 

0'Brien[  .5] 

3.7198 

2 

100 

0.0276557 

Brown-Forsythe 

2.6112 

2 

100 

0.0784510 

Levene 

3.6981 

2 

100 

0.0282189 

Bartlett 

14.6636 

2 

• 

0.0000004 

Welch  Anova  testing  Heans  Equal,  allowing  Std's  Not  Equal 

r  Ratio  or  NUB  or  sen  Prob>r 
2.9995  2  44.718  0.0599 


lAFPFP  soeans  1st  Quarter,  Air  Force,  Production,  Fixed  Price 
lAPFP  means  1st  Quarter,  Rxmy,  Production,  Fixed  Price 
INPFP  means  1st  Quarter,  Navy,  Production,  Fixed  Price 


that  th<  Vari«BC««  are  Kqual  ] 


Level  Count 

Std  Dev 

NeanRbsOif 

to 

Mean  NeanAbsOif 

to  Median 

lAFPFF 

46 

17.14911 

13.26665 

12.40304 

lAPFP 

29 

18.35469 

12.92076 

12.74759 

INPFP 

28 

33.08671 

21.84883 

20.26179 

Test 

r  Ratio 

OF  Nun  or 

Oen 

Prob>P 

0’Brien[ .5] 

2.1662 

2 

100 

0.1199581 

Brown-Forsythe 

1.7987 

2 

100 

0.1708218 

Levene 

2.9917 

2 

100 

0.0547128 

Bartlett 

8.9397 

2 

• 

0.0001311 

Welch  Anova  testing  Means  Equal,  allowing  Std's  Not  Equal 
r  Ratio  or  Nu  or  oen  rrob>r 
4.2796  2  52.678  0.0190 


lAFPFP  means  1st  Quarter,  Air  Force,  Production,  Fixed  Price 
lAPFP  means  1st  Quarter,  Arn^,  Production,  Fixed  Price 
INPFP  mesuis  1st  Quarter,  Navy,  Production,  Fixed  Price 
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[Taata  that  tha  Varlaneas  ara  Kqual 


Laval  Count 

Std  Dev 

NeanAbsOif 

to 

Naan  NaanAbsOif 

to  Madian 

2AFPFP 

58 

11.23464 

5.614203 

5.574655 

2APFP 

31 

11.24569 

8.018273 

7.709677 

2HPFP 

47 

10.64134 

7.171236 

7.123191 

East 

F  Ratio 

OF  NOB  DF 

Dan 

Frob>F 

0'Brian[ .5] 

0.0198 

2 

133 

0.9803550 

Brown*-Forsythe 

0.7179 

2 

133 

0.4896406 

Lavana 

0.8860 

2 

133 

0.4147157 

Bartlett 

0.0877 

2 

• 

0.9160201 

Halch  Anova  tasting  Means  Equal,  allowing  Std's  Not  Equal 
r  Ratio  or  Rub  OF  Dan  Prob>r 
3.2644  2  76.094  0.0436 


2AFPFP  means  2nd  Quarter,  Air  Force,  Production,  Fixed  Price 
2APFP  meeuis  2nd  Quarter,  Army,  Production,  Fixed  Price 
2NPFP  means  2nd  Quarter,  Navy,  Production,  Fixed  Price 
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